[bug#41015] [PATCH] (incomplete) F2FS support

[bug#41015] [PATCH] (incomplete) F2FS support

[raingloom]

[-- Attachment #1: Type: text/plain, Size: 1265 bytes --]

I've been working on this on and off for a while, but can't finish the
last steps, so I'm sending it as is.

What works:
installing to and booting from a partition specified by UUID

What seemed to work but then didn't:
installing to a partition specified by its label. i tried to do the
math, but for some reason it still can't find it. I'm pretty sure it
worked at some point.

What definitely doesn't work:
file system check on boot. I tried and tried to create a statically
linked fsck and I've concluded that it's impossible and that I hate
linkers. Someone else can figure it out, because I've given up.
This means that the init will look for it but won't find it. You will
be thrown into a rescue shell, but if you exit it with `,q`, the init
process continues without any issue.


I used this to set Guix up on my desktop machine on an SSD and
other than the issues I've mentioned, it seems to be working fine.


I'm hoping someone will pick up the work where I left it. If not, I may
revisit it once I've learned enough about how f2fs-tools is built. But I
can't say that I'm eager to devote more of my life to that.


About the patches:
0001-0005 are what you need for a working system, I included the rest
to help anyone who wishes to continue the work.

[-- Attachment #2: 0001-file-system-Add-support-for-F2FS.patch --]
[-- Type: text/x-patch, Size: 4268 bytes --]

From 383d8536b748786b2072f5f61e1e24d0c9b4585d Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Fri, 3 Apr 2020 02:32:23 +0200
Subject: [PATCH 1/8] file-system: Add support for F2FS.

* gnu/build/file-systems.scm (%f2fs-endianness): New syntax.
(f2fs-superblock?, read-f2fs-superblock, f2fs-superblock-uuid)
(f2fs-superblock-volume-name, check-f2fs-file-system): New procedures.
(%partition-label-readers, %partition-uuid-readers, check-file-system): Register them.
---
 gnu/build/file-systems.scm | 61 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 59 insertions(+), 2 deletions(-)

diff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scm
index 902563b219..3742252421 100644
--- a/gnu/build/file-systems.scm
+++ b/gnu/build/file-systems.scm
@@ -336,6 +336,58 @@ if DEVICE does not contain a JFS file system."
     (2 'reboot-required)
     (_ 'fatal-error)))
 
+\f
+;;;
+;;; F2FS (Flash-Friendly File System)
+;;;
+
+;;; https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git/tree/include/linux/f2fs_fs.h
+;;; (but using xxd proved to be simpler)
+
+(define-syntax %f2fs-endianness
+  ;; Endianness of F2FS file systems
+  (identifier-syntax (endianness little)))
+
+;; F2FS actually stores two adjacent copies of the superblock.
+;; should we read both?
+(define (f2fs-superblock? sblock)
+  "Return #t when SBLOCK is an F2FS superblock."
+  (let ((magic (bytevector-u32-ref sblock 0 %f2fs-endianness)))
+    (= magic #xF2F52010)))
+
+(define (read-f2fs-superblock device)
+  "Return the raw contents of DEVICE's F2FS superblock as a bytevector, or #f
+if DEVICE does not contain an F2FS file system."
+  (read-superblock device
+                   ;; offset of magic in first copy
+                   #x400
+                   ;; difference between magic of second
+                   ;; and first copies
+                   (- #x1400 #x400)
+                   f2fs-superblock?))
+
+(define (f2fs-superblock-uuid sblock)
+  "Return the UUID of F2FS superblock SBLOCK as a 16-byte bytevector."
+  (sub-bytevector sblock
+                  (- (+ #x460 12)
+                     ;; subtract superblock offset
+                     #x400)
+                  16))
+
+(define (f2fs-superblock-volume-name sblock)
+  "Return the volume name of SBLOCK as a string of at most 16 characters, or
+#f if SBLOCK has no volume name."
+  (utf16->string (sub-bytevector sblock (- (+ #x470 12) #x400) 512) %f2fs-endianness))
+
+(define (check-f2fs-file-system device)
+  "Return the health of a F2FS file system on DEVICE."
+  (match (status:exit-val
+          (system* "fsck.f2fs" "-p" device))
+    ;; 0 and -1 are the only two possibilities
+    ;; (according to the manpage)
+    (0 'pass)
+    (_ 'fatal-error)))
+
 \f
 ;;;
 ;;; LUKS encrypted devices.
@@ -472,7 +524,9 @@ partition field reader that returned a value."
         (partition-field-reader read-fat16-superblock
                                 fat16-superblock-volume-name)
         (partition-field-reader read-jfs-superblock
-                                jfs-superblock-volume-name)))
+                                jfs-superblock-volume-name)
+        (partition-field-reader read-f2fs-superblock
+                                f2fs-superblock-volume-name)))
 
 (define %partition-uuid-readers
   (list (partition-field-reader read-iso9660-superblock
@@ -486,7 +540,9 @@ partition field reader that returned a value."
         (partition-field-reader read-fat16-superblock
                                 fat16-superblock-uuid)
         (partition-field-reader read-jfs-superblock
-                                jfs-superblock-uuid)))
+                                jfs-superblock-uuid)
+        (partition-field-reader read-f2fs-superblock
+                                f2fs-superblock-uuid)))
 
 (define read-partition-label
   (cut read-partition-field <> %partition-label-readers))
@@ -582,6 +638,7 @@ were found."
      ((string-prefix? "btrfs" type) check-btrfs-file-system)
      ((string-suffix? "fat" type) check-fat-file-system)
      ((string-prefix? "jfs" type) check-jfs-file-system)
+     ((string-prefix? "f2fs" type) check-f2fs-file-system)
      ((string-prefix? "nfs" type) (const 'pass))
      (else #f)))
 
-- 
2.26.2


[-- Attachment #3: 0002-gnu-Add-f2fs-tools-static.patch --]
[-- Type: text/x-patch, Size: 955 bytes --]

From 26208230e82cad199ec45707d156e9c23d43d04c Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Fri, 3 Apr 2020 03:59:52 +0200
Subject: [PATCH 2/8] gnu: Add f2fs-tools-static

* gnu/packages/file-systems.scm (f2fs-tools/static): New public variable.
---
 gnu/packages/linux.scm | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 701df764cd..fc8afbb90c 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -4469,6 +4469,15 @@ disks and SD cards.  This package provides the userland utilities.")
     (inputs
      `(("libuuid" ,util-linux)))))
 
+(define-public f2fs-tools/static
+  (static-package
+   (package
+     (inherit f2fs-tools)
+     (name "f2fs-tools-static")
+     (inputs
+      `(("libuuid:static" ,util-linux "static")
+        ,@(package-inputs f2fs-tools))))))
+
 (define-public freefall
   (package
     (name "freefall")
-- 
2.26.2


[-- Attachment #4: 0003-gnu-Add-f2fs-fsck-static.patch --]
[-- Type: text/x-patch, Size: 2048 bytes --]

From 6fff1095c768b5863e6de45bb9b90aead77e121f Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Fri, 3 Apr 2020 04:18:26 +0200
Subject: [PATCH 3/8] gnu: Add f2fs-fsck-static

* gnu/packages/linux.scm (f2fs-fsck/static): New public variable.
---
 gnu/packages/linux.scm | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index fc8afbb90c..5b752eeef5 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -4478,6 +4478,38 @@ disks and SD cards.  This package provides the userland utilities.")
       `(("libuuid:static" ,util-linux "static")
         ,@(package-inputs f2fs-tools))))))
 
+(define-public f2fs-fsck/static
+  (package
+    (name "f2fs-fsck-static")
+    (version (package-version f2fs-tools/static))
+    (source #f)
+    (build-system trivial-build-system)
+    (arguments
+     `(#:modules ((guix build utils))
+       #:builder
+       (begin
+         (use-modules (guix build utils)
+                      (ice-9 ftw)
+                      (srfi srfi-26))
+         (let* ((f2fs-tools (assoc-ref %build-inputs "f2fs-tools-static"))
+                (fsck "fsck.f2fs")
+                (out (assoc-ref %outputs "out"))
+                (sbin (string-append out "/sbin")))
+           (mkdir-p sbin)
+           (with-directory-excursion sbin
+             (install-file (string-append f2fs-tools "/sbin/" fsck)
+                           ".")
+             (remove-store-references fsck)
+             (chmod fsck #o555))
+           #t))))
+    (inputs
+     `(("f2fs-tools-static" ,f2fs-tools/static)))
+    (home-page (package-home-page f2fs-tools/static))
+    (synopsis "Statically-linked fsck.f2fs command from f2fs-tools")
+    (description "This package provides statically-linked fsck.f2fs command taken
+from the f2fs-tools package. It is meant to be used in initrds.")
+    (license (package-license f2fs-tools/static))))
+
 (define-public freefall
   (package
     (name "freefall")
-- 
2.26.2


[-- Attachment #5: 0004-linux-initrd-Add-support-for-F2FS.patch --]
[-- Type: text/x-patch, Size: 1445 bytes --]

From 0ea3a259b2f89b7a6a6962156ffaffec9d389c99 Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Fri, 3 Apr 2020 07:00:41 +0200
Subject: [PATCH 4/8] =?UTF-8?q?linux-initrd:=20Add=20support=20for=20F2FS.?=
 =?UTF-8?q?=20*=20gnu/system/linux-initrd.scm=20(file-system-packages):=20?=
 =?UTF-8?q?Add=20f2fs-fsck/static.=20(file-system-type-modules):=20Add=20?=
 =?UTF-8?q?=E2=80=98f2fs=E2=80=99=20module.?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

---
 gnu/system/linux-initrd.scm | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index dcc9b6b937..6a1840dbf6 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -245,6 +245,9 @@ FILE-SYSTEMS."
           '())
     ,@(if (find (file-system-type-predicate "jfs") file-systems)
           (list jfs_fsck/static)
+          '())
+    ,@(if (find (file-system-type-predicate "f2fs") file-systems)
+          (list f2fs-fsck/static)
           '())))
 
 (define-syntax vhash                              ;TODO: factorize
@@ -275,6 +278,7 @@ FILE-SYSTEMS."
                     ("btrfs" => '("btrfs"))
                     ("iso9660" => '("isofs"))
                     ("jfs" => '("jfs"))
+                    ("f2fs" => '("f2fs" "crc32_generic"))
                     (else '())))
 
 (define (file-system-modules file-systems)
-- 
2.26.2


[-- Attachment #6: 0005-disable-fsck.f2fs.patch --]
[-- Type: text/x-patch, Size: 973 bytes --]

From 4b8156682cae49df79ef992a55f951256163a6fa Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Fri, 17 Apr 2020 00:51:28 +0200
Subject: [PATCH 5/8] disable fsck.f2fs

---
 gnu/system/linux-initrd.scm | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 6a1840dbf6..349c143298 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -246,9 +246,10 @@ FILE-SYSTEMS."
     ,@(if (find (file-system-type-predicate "jfs") file-systems)
           (list jfs_fsck/static)
           '())
-    ,@(if (find (file-system-type-predicate "f2fs") file-systems)
-          (list f2fs-fsck/static)
-          '())))
+    ;; ,@(if (find (file-system-type-predicate "f2fs") file-systems)
+    ;;       (list f2fs-fsck/static)
+    ;;       '())
+    ))
 
 (define-syntax vhash                              ;TODO: factorize
   (syntax-rules (=>)
-- 
2.26.2


[-- Attachment #7: 0006-gnu-f2fs-tools-static-try-only-using-static-util-lin.patch --]
[-- Type: text/x-patch, Size: 842 bytes --]

From 3faa9923e70416d27eba822d423c5a5007c6df3d Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Sun, 19 Apr 2020 12:30:55 +0200
Subject: [PATCH 6/8] gnu: f2fs-tools-static: try only using static util-linux

---
 gnu/packages/linux.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index ab74380dd6..db7cf03b3e 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -4571,8 +4571,7 @@ disks and SD cards.  This package provides the userland utilities.")
      (inherit f2fs-tools)
      (name "f2fs-tools-static")
      (inputs
-      `(("libuuid:static" ,util-linux "static")
-        ,@(package-inputs f2fs-tools))))))
+      `(("libuuid:static" ,util-linux "static"))))))
 
 (define-public f2fs-fsck/static
   (package
-- 
2.26.2


[-- Attachment #8: 0007-add-temporary-debugger-breakpoint-in-f2fs-tools-stat.patch --]
[-- Type: text/x-patch, Size: 1141 bytes --]

From 4e01a024dd8c9de3affb6a0f628fce7325ef07cc Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Mon, 20 Apr 2020 14:57:33 +0200
Subject: [PATCH 7/8] add temporary debugger breakpoint in f2fs-tools-static

---
 gnu/packages/linux.scm | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 9b56e84477..c0fac7b6e5 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -4570,8 +4570,18 @@ disks and SD cards.  This package provides the userland utilities.")
    (package
      (inherit f2fs-tools)
      (name "f2fs-tools-static")
+     (arguments
+      '(#:make-flags (list "CFLAGS=-v")
+        #:phases
+        (modify-phases %standard-phases
+          (add-before 'install 'fail
+            (lambda _
+              (error "fail on purpose"))))))
      (inputs
-      `(("libuuid:static" ,util-linux "static"))))))
+      `(,@(package-inputs f2fs-tools)
+        ("libuuid:static" ,util-linux "static")
+        ("glibc:static" ,glibc "static")
+        )))))
 
 (define-public f2fs-fsck/static
   (package
-- 
2.26.2


[-- Attachment #9: 0008-trying-the-original-approach-glibc-static-adding-a-b.patch --]
[-- Type: text/x-patch, Size: 1504 bytes --]

From ab44403a7563fe6ca3edf4d1b7929a8d7d12429a Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@protonmail.com>
Date: Thu, 23 Apr 2020 04:14:37 +0200
Subject: [PATCH 8/8] trying the original approach + glibc static + adding a
 "breakpoint"

---
 gnu/packages/linux.scm | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index c0fac7b6e5..801ffbdf88 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -4571,17 +4571,18 @@ disks and SD cards.  This package provides the userland utilities.")
      (inherit f2fs-tools)
      (name "f2fs-tools-static")
      (arguments
-      '(#:make-flags (list "CFLAGS=-v")
-        #:phases
-        (modify-phases %standard-phases
-          (add-before 'install 'fail
-            (lambda _
-              (error "fail on purpose"))))))
+      (substitute-keyword-arguments (package-arguments f2fs-tools)
+        ((#:make-flags flags)
+         `(cons* "CFLAGS=-v" ,flags))
+        ((#:phases phases)
+         `(modify-phases ,phases
+            (add-before 'install 'fail
+              (lambda _
+                (error "fail on purpose")))))))
      (inputs
-      `(,@(package-inputs f2fs-tools)
-        ("libuuid:static" ,util-linux "static")
+      `(("libuuid:static" ,util-linux "static")
         ("glibc:static" ,glibc "static")
-        )))))
+        ,@(package-inputs f2fs-tools))))))
 
 (define-public f2fs-fsck/static
   (package
-- 
2.26.2

[bug#41015] [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

Hi,

thanks!

I know exactly what you mean--I've had some "fun" preparing initrds too.

Pushed the first patch to guix master as commit 23b37c3d40d497cc6f07437ab26ab10e60fb6e09
and a variant of your fourth patch to guix master as commit 27efeef3068bfa21011ea509e21efcbe8c353b5f
so you should be able to at least boot.

Note:

https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git/tree/include/linux/f2fs_fs.h says volume name can have up to 512 chars.

Your comment says 16, but your code uses 512.  I've fixed comment to say 512.

fsck progress so far: fsck.f2fs links to libuuid dynamically...

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bug#41015] [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 72 bytes --]

fsck progress so far:

Everything except mkfs.f2fs links statically.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bug#41015] [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 230 bytes --]

Pushed that version of f2fs-tools-static to guix master as commit
807986a55fc2849d6986efb79f9a015cf4132e09 anyway since it is a problem
in our util-linux package.

See also https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41019

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bug#41015] [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 83 bytes --]

Pushed patch 3 to guix master as commit 2dd2210bf9065c9dcc3ec5412776a88b4df0201b.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bug#41015] [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

On Sat, 2 May 2020 02:48:00 +0200
raingloom <raingloom@riseup.net> wrote:

> What seemed to work but then didn't:
> installing to a partition specified by its label. i tried to do the
> math, but for some reason it still can't find it. I'm pretty sure it
> worked at some point.

Please try:

$ guile --no-auto-compile -L .
((@@ (gnu build file-systems) f2fs-superblock-volume-name)
 ((@@ (gnu build file-systems) read-f2fs-superblock) "your-device"))


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bug#41015] [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 98 bytes --]

Correction:

Pushed patch 3 to guix master as commit a5cf52d3aa31717a8ae0c2cbbc4b848266ed7818.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bug#41015] [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 150 bytes --]

I've tried it now--it wasn't honoring the UTF-16 NUL termination.
I've fixed it now in guix master commit f73f4b3a2d7a313a6cb1667bd69205ea4b09f57c.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#41015: [PATCH] (incomplete) F2FS support

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Sat, 2 May 2020 11:56:54 +0200
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> See also https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41019

Pushed a workaround for that bug to guix master in commit
da09c63e78ebebeabb347f483d7284b87ff51c2f
and enabled f2fs fsck in commit 33eab4a10dcd2a6580f168f18455df1d4653d14b.

Added system installation test in commit af7a8e718dddd12c3dddf92bfce2f79471fa4a0d.

Added workaround for mising support for UTF-16 in the initrd's guile in
commit bb357c509e1c017e1fef5aa5f4d05beea0c25157.

That concludes this patch series.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]