[bug#41804] ISO image & HFS+ tree

[bug#41804] ISO image & HFS+ tree

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]


Hello,

When running the "gui-installed-desktop-os-encrypted" test which
produces a big ISO9660 image (~4GiB), the following error occurs:

--8<---------------cut here---------------start------------->8---
xorriso : UPDATE :  434919 files added in 18 seconds
libisofs: FAILURE : HFS+ map nodes aren't implemented
libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file names
--8<---------------cut here---------------end--------------->8---

I asked Thomas Schmitt (of GNU Xorriso) about it. It seems that
producing an HFS+ tree could fail if a directory contains more than
30720 files.

HFS+ seems to be used to work-around filesystem limits that we don't
have on Intel architectures (see Thomas kind answer below).

Here's a patch that disables HFS+ tree production, and the solves the
problem above.

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-system-image-Do-not-produce-an-HFS-tree-when-buildin.patch --]
[-- Type: text/x-diff, Size: 1409 bytes --]

From 604adfe81615ea05905fc73d62fba59a19437cdf Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe@gnu.org>
Date: Thu, 11 Jun 2020 15:25:02 +0200
Subject: [PATCH] system: image: Do not produce an HFS tree when building an
 ISO.

Producing an HFS tree will fail if a directory has more that 30720 files. The
xorriso error message is:

xorriso : UPDATE :  434919 files added in 18 seconds
libisofs: FAILURE : HFS+ map nodes aren't implemented
libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file names

Use "mbr_only" mode to make sure that no HFS+ tree are generated.

* gnu/system/image.scm (system-image): Set MKRESCUE_SED_MODE to "mbr_only".
---
 gnu/system/image.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index a0e6bf31f1..9e47bcc508 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -545,7 +545,7 @@ image, depending on IMAGE format."
                              #:inputs `(("system" ,os)
                                         ("bootcfg" ,bootcfg))
                              #:grub-mkrescue-environment
-                             '(("MKRESCUE_SED_MODE" . "mbr_hfs")))))))
+                             '(("MKRESCUE_SED_MODE" . "mbr_only")))))))
 
 (define (find-image file-system-type)
   "Find and return an image that could match the given FILE-SYSTEM-TYPE.  This
-- 
2.26.2


[-- Attachment #3: hfs.raw --]
[-- Type: application/octet-stream, Size: 4649 bytes --]

X-Gnus-Coding-System: -*- coding: utf-8; -*-

From: "Thomas Schmitt" <scdbackup@gmx.net>
Subject: Re: "Too much files to mangle" error
To: bug-xorriso@gnu.org
Cc: othacehe@gnu.org
Date: Thu, 11 Jun 2020 13:38:49 +0200 (2 hours, 44 minutes, 51 seconds ago)

Hi,

Mathieu Othacehe wrote:
> xorriso : UPDATE :  434919 files added in 18 seconds
> libisofs: FAILURE : HFS+ map nodes aren't implemented
> libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file names

Ouchers. That's in the most moldy cellar dimensions of libisofs.

The "too much files to mangle" error can occur in the ISO 9660 tree,
the HFS+ tree, the ISO 9660:1999 tree, and the Joliet tree.
In this case the HFS+ generator is so friendly to also issue a message
before it bails out with ISO_MANGLE_TOO_MUCH_FILES.

My diagnosis attempt below brings me to the theory that a directory has
more than 30720 files and that the HFS+ implementation in libisofs cannot
stand this.


> I'm trying to generate an ISO image for GNU Guix
> [...]
> Is this possible to get around this limit?

Try to suppress production of HFS+. AFAIK, guix uses grub-mkrescue, which
issues xorrisofs option -hfsplus quite unconditionally.
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n718

If you cannot keep your grub-mkrescue from doing this, then i can offer
the script grub-mkrescue-sed.sh which can be submitted to grub-mkrescue
by option

  --xorriso=/...absolute.../...path.../grub-mkrescue-sed.sh

The script usage is described in the large comment at its start.
It manipulates the xorriso options from grub-mkrescue and then uses
them in a run of xorriso.

The default setting will change the xorrisofs options so that no HFS+
and no GPT will be generated. I.e. EFI will be served by a MBR partition
of type 0xef rather than a GPT partition.
It is advised to add xorrisofs option
  -partition_offset 16
to the grub-mkrescue options in order to get the EFI partition claimed
as part of the ISO filesystem and at the same time get a mountable ISO
partition. (Your quite large directory tree will be recorded twice in
the ISO, i fear. So the result will grow by 100 MB or more.)

If you export environment variable
  MKRESCUE_SED_MODE=gpt_appended
before the grub-mkrescue run with grub-mkrescue-sed.sh, then you get GPT
and unconditionally -partition_offset 16.

If you don't have the script from a xorriso or libisoburn installation,
you can download it from

  https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh

(It should not be too hard to add a MKRESCUE_SED_MODE "no_hfs" to the
script or to create an own pseudo-xorriso iscript which passes all its
aruments to xorriso, except the problematic "-hfsplus".)


-------------------------------------------------------------------------
Now for a diagnosis:

HFS+ in libisofs is a contribution by Vladimir Serbinenko who was back then
maintainer of GRUB and is still to see from time to time at grub-devel
mailing list.

Mangling is a collision prevention mechanism for filesystem trees which
have a more restricted name length or character set than X/Open demands.
Names get changed by some scheme that has its natural limits.

But Vladimir seems to have re-used the error code for something different
than mangling:

  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus.c
line 1791 has:

    if (target->hfsp_nnodes > (cat_node_size - 0x100) * 8)
      {
        iso_msg_submit(target->image->id, ISO_MANGLE_TOO_MUCH_FILES, 0,
                        "HFS+ map nodes aren't implemented");
        ret = ISO_MANGLE_TOO_MUCH_FILES;
        goto ex;
      }

This looks more like a plain overflow.

target->hfsp_nnodes is a counter which i have difficulties to understand:

      target->hfsp_levels[level].level_size++;
      target->hfsp_nnodes += target->hfsp_levels[level].level_size;

cat_node_size is set by

    target->hfsp_cat_node_size = 2 * target->opts->hfsp_block_size;
    ...
    cat_node_size = target->hfsp_cat_node_size;

grub-mkrescue sets the HFS+ blocksize to 2048.
My rough theory is that a HFS+ directory cannot have more than
   (4096 - 256) * 8 = 30720
file names. There seems to be a workaround "HFS+ map nodes". But the HFS+
code in libisofs reports not to have it implemented.


I will ask at grub-devel mailing list whether my theory is correct and
whether HFS+ is still important enough to be used for I386 and X86_64
systems. (It could be essential for POWERPC_IEEE1275.)
But except Vladimir nobody else there has an opinion about grub-mkrescue
and Vladimir does not appear often any more.


Have a nice day :)

Thomas

[bug#41804] ISO image & HFS+ tree

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

Hi Mathieu,

thanks for On Thu, 11 Jun 2020 16:24:48 +0200
Mathieu Othacehe <othacehe@gnu.org> wrote:

> Here's a patch that disables HFS+ tree production, and the solves the
> problem above.

Thanks for the investigation and fix!

What do you think about putting the comment into the source code?

;; Producing an HFS tree via "mbr_hfs" would fail if a directory has more than
;; 30720 files. The xorriso error message is:
;; xorriso : UPDATE :  434919 files added in 18 seconds
;; libisofs: FAILURE : HFS+ map nodes aren't implemented
;; libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file
;; names.
;; So we use "mbr_only" mode to make sure that no HFS+ tree is generated.
'(("MKRESCUE_SED_MODE" . "mbr_only")))))))

You can remove it from the commit message, or not (I would).

In any case I'd rather not have to read the source code *and* all commit
messages once some problem appears.

I think it's as important to know what not to do as it is to know what is
possible--especially for something as obscure as that.

So I for one would welcome a comment in the source code of what already
caused problems, even re-adding the problem as a comment.

Otherwise LGTM!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#41804: ISO image & HFS+ tree

[Mathieu Othacehe]

Hey Danny,

> In any case I'd rather not have to read the source code *and* all commit
> messages once some problem appears.
>
> I think it's as important to know what not to do as it is to know what is
> possible--especially for something as obscure as that.
>
> So I for one would welcome a comment in the source code of what already
> caused problems, even re-adding the problem as a comment.

Yes, you're right, I followed your advice and added some details as
comments. Thanks for reviewing, pushed!

Mathieu