Modifying squashfs usage for `guix pack` for supporting singularity

Modifying squashfs usage for `guix pack` for supporting singularity

[Josh Marshall]

I ran into some trouble today when trying to package an image for singularity on CentOS7.  nxck suggested I go into the cdoe and try adding the "-noappend" option to mksquashfs in 'guix/guix/scripts/pack.scm'.  While looking at it, it seems that it performs a multistep process to modify the squashfs image, which kind of goes against the design of the utility.  A SO page ( https://unix.stackexchange.com/a/459372 ) suggested the full hierarchy be created and then added in a single invocation.  Does anyone have any opinions on such a change or useful insights?
---

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

Re: Modifying squashfs usage for `guix pack` for supporting singularity

[Josh Marshall]

It also appears the `-mkfs-time 0`, `-all-time 0`, and `-UTC` arguments have been missing.

Docker and Singularity havebeen moving towards support of the Open Containers Initative, which prompts the discussion of if the docker and sinfularity (squashfs) pack options should be depricated in favor of this standard: https://github.com/opencontainers/image-spec

________________________________________
From: Josh Marshall
Sent: Wednesday, March 4, 2020 4:00 PM
To: guix-devel@gnu.org
Subject: Modifying squashfs usage for `guix pack` for supporting singularity

I ran into some trouble today when trying to package an image for singularity on CentOS7.  nxck suggested I go into the cdoe and try adding the "-noappend" option to mksquashfs in 'guix/guix/scripts/pack.scm'.  While looking at it, it seems that it performs a multistep process to modify the squashfs image, which kind of goes against the design of the utility.  A SO page ( https://unix.stackexchange.com/a/459372 ) suggested the full hierarchy be created and then added in a single invocation.  Does anyone have any opinions on such a change or useful insights?
---

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

Re: Modifying squashfs usage for `guix pack` for supporting singularity

[Ludovic Courtès]

Hi Josh,

Apologies for the delay!  (In general, it’s probably safer to report
bugs to bug-guix@gnu.org because they’re tracked and are less likely to
fall through the cracks.)

Josh Marshall <Josh.Marshall@jax.org> skribis:

> I ran into some trouble today when trying to package an image for singularity on CentOS7.  nxck suggested I go into the cdoe and try adding the "-noappend" option to mksquashfs in 'guix/guix/scripts/pack.scm'.  While looking at it, it seems that it performs a multistep process to modify the squashfs image, which kind of goes against the design of the utility.  A SO page ( https://unix.stackexchange.com/a/459372 ) suggested the full hierarchy be created and then added in a single invocation.  Does anyone have any opinions on such a change or useful insights?

[...]

> It also appears the `-mkfs-time 0`, `-all-time 0`, and `-UTC` arguments have been missing.

Could you explain what error you got and what makes you think these
options were missing?

IME, the images produced by ‘guix pack -f squashfs’ work fine with
Singularity 2.x and 3.x.

> Docker and Singularity havebeen moving towards support of the Open Containers Initative, which prompts the discussion of if the docker and sinfularity (squashfs) pack options should be depricated in favor of this standard: https://github.com/opencontainers/image-spec

In fact, ‘guix pack -f docker’ produces an OCI image, IIUC.

Singularity 3.x introduced the Singularity Image Format (SIF), which is
yet another format.  AIUI, SIF is the recommended format for Singularity
3.x, isn’t it?

Thanks,
Ludo’.

Re: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

[Josh Marshall]

[-- Attachment #1: Type: text/plain, Size: 5462 bytes --]

Thanks for getting back to me on this.

For the options I mentioned in the email, and the one not yet tracked in email since it is only in IRC is "--no-recovery".  These are what stood out to me when reading the `mksquashfs` documentation ( https://github.com/plougher/squashfs-tools/blob/master/USAGE ) and looking at `guix/scripts/pack.scm'.  The knowledge you're saying about the `guix pack` options is not communicated by `guix pack --list-formats`, which yields

```
$ guix pack --list-formats
The supported formats for 'guix pack' are:

  tarball       Self-contained tarball, ready to run on another machine
  squashfs      Squashfs image suitable for Singularity
  docker        Tarball ready for 'docker load'
```

All this seems to be tangential to the fact that `guix pack --format=squashfs ...` fails on CentOS7 for an undescribed reason:
```
$ guix pack --format=squashfs bash-minimal
substitute: /gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
substitute: /gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv
   /gnu/store/84vj4wgqm46rws1v73h9gc3hr29xdrs6-singularity-environment.sh.drv
   /gnu/store/iwrrg2knk45h4fps88jqfg92ff1x44zi-profile.drv
The following profile hooks will be built:
   /gnu/store/dgh995fyb41gr3il8r2770j2hlf22nzv-fonts-dir.drv
   /gnu/store/h5dr8a6vpzrk80wrjz586k3axpz9bhqa-manual-database.drv
   /gnu/store/svyngf4v16q3j8lgsql508573mvc64my-info-dir.drv
   /gnu/store/svyydyprj7id6gfz6hsifaf7v4g02brj-ca-certificate-bundle.drv
building CA certificate bundle...
building fonts directory...
building directory of Info manuals...
building database for manual pages...
building /gnu/store/iwrrg2knk45h4fps88jqfg92ff1x44zi-profile.drv...
building /gnu/store/84vj4wgqm46rws1v73h9gc3hr29xdrs6-singularity-environment.sh.drv...
/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
building /gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv...
\builder for `/gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv' failed with exit code 1
build of /gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv failed
View build log at '/var/log/guix/drvs/sa/ar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv.bz2'.
guix pack: error: build of `/gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv' failed
```

According to the OCI description, Singularity's .sif format could be skipped, but at my work it is /the/ standard format.  It sounds like there should be in program changes to describe supported container formats, and which systems support which.  This is in addition to getting squashfs container format working on CentOS7, and applying some changes to more correctly use `mksquashfs`.

________________________________________
From: Ludovic Courtès <ludovic.courtes@inria.fr>
Sent: Thursday, March 12, 2020 9:51 AM
To: Josh Marshall
Cc: guix-devel@gnu.org
Subject: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

Hi Josh,

Apologies for the delay!  (In general, it’s probably safer to report
bugs to bug-guix@gnu.org because they’re tracked and are less likely to
fall through the cracks.)

Josh Marshall <Josh.Marshall@jax.org> skribis:

> I ran into some trouble today when trying to package an image for singularity on CentOS7.  nxck suggested I go into the cdoe and try adding the "-noappend" option to mksquashfs in 'guix/guix/scripts/pack.scm'.  While looking at it, it seems that it performs a multistep process to modify the squashfs image, which kind of goes against the design of the utility.  A SO page ( https://unix.stackexchange.com/a/459372 ) suggested the full hierarchy be created and then added in a single invocation.  Does anyone have any opinions on such a change or useful insights?

[...]

> It also appears the `-mkfs-time 0`, `-all-time 0`, and `-UTC` arguments have been missing.

Could you explain what error you got and what makes you think these
options were missing?

IME, the images produced by ‘guix pack -f squashfs’ work fine with
Singularity 2.x and 3.x.

> Docker and Singularity havebeen moving towards support of the Open Containers Initative, which prompts the discussion of if the docker and sinfularity (squashfs) pack options should be depricated in favor of this standard: https://github.com/opencontainers/image-spec

In fact, ‘guix pack -f docker’ produces an OCI image, IIUC.

Singularity 3.x introduced the Singularity Image Format (SIF), which is
yet another format.  AIUI, SIF is the recommended format for Singularity
3.x, isn’t it?

Thanks,
Ludo’.
---

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

[-- Attachment #2: ar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv.bz2 --]
[-- Type: application/x-bzip, Size: 1218 bytes --]

Re: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

[Ludovic Courtès]

Hi Josh,

Josh Marshall <Josh.Marshall@jax.org> skribis:

> Thanks for getting back to me on this.
>
> For the options I mentioned in the email, and the one not yet tracked
> in email since it is only in IRC is "--no-recovery".  These are what
> stood out to me when reading the `mksquashfs` documentation (
> https://github.com/plougher/squashfs-tools/blob/master/USAGE ) and
> looking at `guix/scripts/pack.scm'.

Perhaps I’m missing something, but it’s still not clear to me why these
options would be needed, especially since images produced with ‘guix
pack -f squashfs’ today appear to work well.

What am I missing?

> The knowledge you're saying about the `guix pack` options is not
> communicated by `guix pack --list-formats`, which yields
>
> ```
> $ guix pack --list-formats
> The supported formats for 'guix pack' are:
>
>   tarball       Self-contained tarball, ready to run on another machine
>   squashfs      Squashfs image suitable for Singularity
>   docker        Tarball ready for 'docker load'
> ```

Right, it just says “docker”, not “OCI”.  The manual is a bit more
verbose:

  https://guix.gnu.org/manual/en/html_node/Invoking-guix-pack.html

What would you suggest adding?

> All this seems to be tangential to the fact that `guix pack --format=squashfs ...` fails on CentOS7 for an undescribed reason:
> ```
> $ guix pack --format=squashfs bash-minimal

[...]

> \builder for `/gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv' failed with exit code 1
> build of /gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv failed
> View build log at '/var/log/guix/drvs/sa/ar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv.bz2'.

Could you email bug-guix@gnu.org with the command above, the output of
‘guix describe’, and the contents of
/var/log/guix/drvs/sa/ar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv.bz2 ?

CentOS 7 runs a very old kernel, so it could be that we’re using some
unsupported feature.

Thanks for your feedback!

Ludo’.

Re: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

[Josh Marshall]

While the images seem to work well in most cases, removing ways for bits to change seems to be in the spirit of guix, and so the "-all-time" and "-mkfs-time" options remove one source of change.  Additionally, without the "--no-recovery" an extra file is made which is not useful to the process, and for my manual testing seemed to cause write or permission issues which could cause failures.  Guix already handles the failure cases in mksquashfs which the recovery file is for and so the redundant work is not helpful, can cause problems, slows down the process, and requires more resources.

As for the information given by `guix pack --list-formats`, I'd change to something like the following:

```
  tarball       Self-contained tarball, ready to run directly on any Linux machine
  squashfs      Container suitable for Singularity 2.x and 3.x
  OCI       OCI compliant tarball ready for 'docker load' and usable by Singularity 3.x
  sif         Unsupported; container preferred by Singularity 3.x

For more information, refer to https://guix.gnu.org/manual/en/html_node/Invoking-guix-pack.html
```

________________________________________
From: Ludovic Courtès <ludovic.courtes@inria.fr>
Sent: Thursday, March 12, 2020 12:39 PM
To: Josh Marshall
Cc: guix-devel@gnu.org
Subject: Re: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

Hi Josh,

Josh Marshall <Josh.Marshall@jax.org> skribis:

> Thanks for getting back to me on this.
>
> For the options I mentioned in the email, and the one not yet tracked
> in email since it is only in IRC is "--no-recovery".  These are what
> stood out to me when reading the `mksquashfs` documentation (
> https://github.com/plougher/squashfs-tools/blob/master/USAGE ) and
> looking at `guix/scripts/pack.scm'.

Perhaps I’m missing something, but it’s still not clear to me why these
options would be needed, especially since images produced with ‘guix
pack -f squashfs’ today appear to work well.

What am I missing?

> The knowledge you're saying about the `guix pack` options is not
> communicated by `guix pack --list-formats`, which yields
>
> ```
> $ guix pack --list-formats
> The supported formats for 'guix pack' are:
>
>   tarball       Self-contained tarball, ready to run on another machine
>   squashfs      Squashfs image suitable for Singularity
>   docker        Tarball ready for 'docker load'
> ```

Right, it just says “docker”, not “OCI”.  The manual is a bit more
verbose:

  https://guix.gnu.org/manual/en/html_node/Invoking-guix-pack.html

What would you suggest adding?

> All this seems to be tangential to the fact that `guix pack --format=squashfs ...` fails on CentOS7 for an undescribed reason:
> ```
> $ guix pack --format=squashfs bash-minimal

[...]

> \builder for `/gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv' failed with exit code 1
> build of /gnu/store/saar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv failed
> View build log at '/var/log/guix/drvs/sa/ar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv.bz2'.

Could you email bug-guix@gnu.org with the command above, the output of
‘guix describe’, and the contents of
/var/log/guix/drvs/sa/ar594fgxibr6vkrpjq2n2l5mpyycpp-squashfs-pack.gz.squashfs.drv.bz2 ?

CentOS 7 runs a very old kernel, so it could be that we’re using some
unsupported feature.

Thanks for your feedback!

Ludo’.
---

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

Re: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

[Ludovic Courtès]

Hi Josh,

Josh Marshall <Josh.Marshall@jax.org> skribis:

> While the images seem to work well in most cases, removing ways for bits to change seems to be in the spirit of guix, and so the "-all-time" and "-mkfs-time" options remove one source of change.  Additionally, without the "--no-recovery" an extra file is made which is not useful to the process, and for my manual testing seemed to cause write or permission issues which could cause failures.  Guix already handles the failure cases in mksquashfs which the recovery file is for and so the redundant work is not helpful, can cause problems, slows down the process, and requires more resources.

Oh, I hadn’t understood that there were reproducibility issues with
‘guix pack -f squashfs’, sorry about that!

Commits 24fb0dc0ab34ebb49509a3d5b4d84d8488670807 and
b829864d747b3b24ef37cafe36e889527b060d4d implement what you suggest.

I can confirm that now something like:

  guix pack -f squashfs sed --rounds=2

passes, which was not the case before.

Thanks!

> As for the information given by `guix pack --list-formats`, I'd change to something like the following:
>
> ```
>   tarball       Self-contained tarball, ready to run directly on any Linux machine
>   squashfs      Container suitable for Singularity 2.x and 3.x
>   OCI       OCI compliant tarball ready for 'docker load' and usable by Singularity 3.x
>   sif         Unsupported; container preferred by Singularity 3.x

We cannot change the format name, for compatibility reasons, but I’ll
double-check that what ‘-f docker’ produces is indeed OCI, and if so,
add it to the description.

> For more information, refer to https://guix.gnu.org/manual/en/html_node/Invoking-guix-pack.html

I don’t think we’re going to add references to the manual in --help and
similar: it’s always implicit that documentation is in the manual.
However, perhaps we could add terminal hyperlinks to make it more
discoverable.

Thanks,
Ludo’.

Re: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

[Josh Marshall]

Glad to have contributed :)

________________________________________
From: Ludovic Courtès <ludovic.courtes@inria.fr>
Sent: Friday, March 13, 2020 12:38 PM
To: Josh Marshall
Cc: guix-devel@gnu.org
Subject: Re: [EXTERNAL]Re: Modifying squashfs usage for `guix pack` for supporting singularity

Hi Josh,

Josh Marshall <Josh.Marshall@jax.org> skribis:

> While the images seem to work well in most cases, removing ways for bits to change seems to be in the spirit of guix, and so the "-all-time" and "-mkfs-time" options remove one source of change.  Additionally, without the "--no-recovery" an extra file is made which is not useful to the process, and for my manual testing seemed to cause write or permission issues which could cause failures.  Guix already handles the failure cases in mksquashfs which the recovery file is for and so the redundant work is not helpful, can cause problems, slows down the process, and requires more resources.

Oh, I hadn’t understood that there were reproducibility issues with
‘guix pack -f squashfs’, sorry about that!

Commits 24fb0dc0ab34ebb49509a3d5b4d84d8488670807 and
b829864d747b3b24ef37cafe36e889527b060d4d implement what you suggest.

I can confirm that now something like:

  guix pack -f squashfs sed --rounds=2

passes, which was not the case before.

Thanks!

> As for the information given by `guix pack --list-formats`, I'd change to something like the following:
>
> ```
>   tarball       Self-contained tarball, ready to run directly on any Linux machine
>   squashfs      Container suitable for Singularity 2.x and 3.x
>   OCI       OCI compliant tarball ready for 'docker load' and usable by Singularity 3.x
>   sif         Unsupported; container preferred by Singularity 3.x

We cannot change the format name, for compatibility reasons, but I’ll
double-check that what ‘-f docker’ produces is indeed OCI, and if so,
add it to the description.

> For more information, refer to https://guix.gnu.org/manual/en/html_node/Invoking-guix-pack.html

I don’t think we’re going to add references to the manual in --help and
similar: it’s always implicit that documentation is in the manual.
However, perhaps we could add terminal hyperlinks to make it more
discoverable.

Thanks,
Ludo’.
---

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.