linux-libre source tarballs

linux-libre source tarballs

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 3984 bytes --]

So, for better or worse, I do a lot with guix on aarch64 platforms and
sometimes even armhf ones. And these platforms are...  often...
quite...  slow...

The way guix does updates of linux-libre sources by downloading upstream
linux tarballs and applying the linux-libre deblobbing and related
scripts to them is really technically excellent.

Pragmatically speaking, on slower platforms this is a huge resource
overhead. So much so that ci.guix.gnu.org *usually* times out when
generating the linux-libre aarch64 tarballs:

  https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic

Same story for "linux-libre".

This means that platforms that are already fairly slow have to build one
or more source tarballs every linux-libre update, which can take several
hours to build the source for a kernel that takes only an hour or less
of compilation time.


I explored some of the alternatives last year:

  https://lists.gnu.org/archive/html/guix-devel/2020-08/msg00089.html

Some of the surrounding threads also go into various issues around this
topic, but I'll summarize what I tried:

* Using linux-libre's git repository. A *bit* slow to download and
  consumed ~1GB of space in /gnu/store, though much faster than running
  the deblobbing scripts. Probably would deduplicate fairly well between
  different versions, but would require re-downloading a lot of
  content. Maybe a more savvy method to make use of this would be
  possible. If a method somewhat like the mirror:// syntax for git
  repositories allowed for a local git cache or a local git proxy, this
  could save a lot of downloading and still get some benefits of
  git. Needs someone to actually do the work, of course...
 
* Using the linux-libre tarballs. May have some issues with long-term
  availability, but fast to download and the deblobbing scripts have
  effectively already been run, saving a lot of local work. There is
  basically support for this in guix already, and was the method used in
  guix before july 2019 1ad9c105c208caa9059924cbfbe4759c8101f6c9.


Recently I've been thinking more about this for aarch64 and armhf; there
are a few things that might help considerably:

* Bump the timeouts on ci.guix.gnu.org for linux-libre so that the
  deblob scripts actually get a chance to finish. Doesn't require much
  change in the guix infrastructure to significantly improve the status
  quo for slower architectures. Might take some tweaking over time to
  find the right timeouts.

* Default to using the fsfla linux-libre tarballs on aarch64 and armhf,
  while doing the full local deblobbing process for other faster
  architectures. This keeps the infrastructure of both methods
  up-to-date and tested, and can be used to do interesting things like
  comparing to make sure the linux-libre tarballs guix generates and the
  fsfla linux-libre tarballs do not contain significant
  differences. This may very well require maintenance overhead by
  generating two different hashes (one for the guix deblobbed tarball,
  and one for the linux-libre deblobbed tarball), and doing things a
  little differently by architecture is a little awkward.

* Produce the guix tarballs in a way that architectures could share the
  resulting tarball. I'm not sure what hoops this would require jumping
  through, something like generate the tarball, upload somewhere, update
  hash. The tarball produced by running the deblob scripts really
  shouldn't be architecture-dependent; re-doing this work on each
  architecture seems suboptimal (though I get is also somewhat a result
  of the functional package management model).


I like the idea of keeping all the infrastructure there to handle what
is currently done now, but would really like to figure out alternatives
at least for aarch64 and armhf.


Thanks for reading so far, please let me know what you think!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: linux-libre source tarballs

[Leo Famulari]

On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote:
> Pragmatically speaking, on slower platforms this is a huge resource
> overhead. So much so that ci.guix.gnu.org *usually* times out when
> generating the linux-libre aarch64 tarballs:
> 
>   https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic

Thanks for letting me know. I didn't know this was happening.

The immediate solution is for me to make sure the tarballs have built
before committing the updates. I already do this for x86_64 and I can
start doing it for aarch64 too.

> * Using linux-libre's git repository.

We could do this. The decision is up to the maintainers, ultimately. I'd
rather we did not because linux-libre is not the "upstream" of the
kernel in a meaningful way (they do not develop or fix bugs), and it's
not "source code" in the GNU sense, which is the "preferred form of the
work for making changes in it":

http://www.gnu.org/licenses/gpl-faq.en.html#GPLOtherThanSoftware

Not to mention that cloning a 1 GB Git repo is rather expensive compared
to downloading a tarball, and thus also suboptimal for the ARM boards
that can't easily deblob. Although if people are building the
linux-libre tarballs, they are also compiling, which is of course way
more expensive.
  
> * Using the linux-libre tarballs. May have some issues with long-term
>   availability, but fast to download and the deblobbing scripts have
>   effectively already been run, saving a lot of local work. There is
>   basically support for this in guix already, and was the method used in
>   guix before july 2019 1ad9c105c208caa9059924cbfbe4759c8101f6c9.

As you say, these tarballs are not available for very long.  I think
this option is not practical for that reason.

> * Bump the timeouts on ci.guix.gnu.org for linux-libre so that the
>   deblob scripts actually get a chance to finish. Doesn't require much
>   change in the guix infrastructure to significantly improve the status
>   quo for slower architectures. Might take some tweaking over time to
>   find the right timeouts.

Will do.

Re: linux-libre source tarballs

[Leo Famulari]

On Sat, May 01, 2021 at 10:45:22PM -0400, Leo Famulari wrote:
> The immediate solution is for me to make sure the tarballs have built
> before committing the updates. I already do this for x86_64 and I can
> start doing it for aarch64 too.

I started building the current derivations "by hand" on the build farm
with a longer max-silent-time.

I'll check in a day or so if it succeeded and how long it took. We can
adjust the packages based on that to avoid "silent time" timeouts even
when I don't pre-build these tarballs.

Re: linux-libre source tarballs

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Sat, May 01, 2021 at 10:45:22PM -0400, Leo Famulari wrote:
>> The immediate solution is for me to make sure the tarballs have built
>> before committing the updates. I already do this for x86_64 and I can
>> start doing it for aarch64 too.
>
> I started building the current derivations "by hand" on the build farm
> with a longer max-silent-time.

For packages we can add a ‘max-silent-time’ property, but there’s
nothing like this for origins.

I wonder if there’s a way we could address it in (gnu ci).

Thoughts?

Ludo’.

Re: linux-libre source tarballs

[Leo Famulari]

On Sun, May 02, 2021 at 11:08:49PM +0200, Ludovic Courtès wrote:
> For packages we can add a ‘max-silent-time’ property, but there’s
> nothing like this for origins.

Oh, right.

> I wonder if there’s a way we could address it in (gnu ci).
> 
> Thoughts?

In the meantime, can we make a 'kernel-updates' Cuirass job again, that
would build a kernel-updates branch on Savannah?

We had one recently for a brief period but I stopped using it, because I
could work more quickly building the tarballs "by hand" (although I've
only been doing it for x86_64).

However, it would be helpful to let Cuirass at least try to build these
derivations, with me checking the results and retrying failures before
pushing kernel-updates to master.

Re: linux-libre source tarballs

[Leo Famulari]

On Sun, May 02, 2021 at 06:26:15PM -0400, Leo Famulari wrote:
> In the meantime, can we make a 'kernel-updates' Cuirass job again, that
> would build a kernel-updates branch on Savannah?

Mistake: I'd want to call the branch wip-kernel-updates, so that it's
rebaseable.

> We had one recently for a brief period but I stopped using it, because I
> could work more quickly building the tarballs "by hand" (although I've
> only been doing it for x86_64).
> 
> However, it would be helpful to let Cuirass at least try to build these
> derivations, with me checking the results and retrying failures before
> pushing kernel-updates to master.

Re: linux-libre source tarballs (disable "deblob-check"?)

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

On Sat, May 01, 2021 at 10:45:22PM -0400, Leo Famulari wrote:
> The immediate solution is for me to make sure the tarballs have built
> before committing the updates. I already do this for x86_64 and I can
> start doing it for aarch64 too.

Well, this is easier said than done, currently.

I was able to build the "arm64-generic" tarball for 5.11.18, but we
actually don't have the capability to build these things reliably at the
moment.

We only have 1 real aarch64 machine (Overdrive 1000) [0] in the build
farm, and it's never idle. The emulated builders are super slow.

We could change (gnu ci) to increase the timeout for these jobs, like
Ludo suggested, but I'm not sure it's worth it? I estimate the Overdrive
would spend at least one day a week just building tarballs, which is
unreasonable IMO.

The problem is that the linux-libre team told us that we shouldn't trust
the linux-libre deblobbing scripts to work and that we have to
"deblob-check" them, which takes a very long time. [1]

I suggest we disable these checks for now, at least for aarch64; have
they ever actually found a problem?

At least for aarch64, the choice is not between "blobby tarball or
non-blobby tarball", but rather between "no tarball or tarball".

[0] I can add another 4 cores (Macchiatobin) to the build farm, maybe
next weekend.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: linux-libre source tarballs

[Alexandre Oliva]

Hello, Leo,

On May  1, 2021, Leo Famulari <leo@famulari.name> wrote:

> it's not "source code" in the GNU sense, which is the "preferred form
> of the work for making changes in it":

That's not accurate.  Linux-libre sources, whether obtained from a git
repository or from a tarball, are just as suitable for kernel
development as any other snapshot.

There is an alternate understanding of source code form taking shape in
some communities that suggests that a version control repository is
somehow more "source"ish than any of the versions held in it.

No doubt the additional versions in it may be useful in some cases, but
it does not follow that other versions and their relationship are part
of "source code" in the GNU sense.

In the GNU sense, any individual snapshot or release meets the
definition of source code.

> Not to mention that cloning a 1 GB Git repo is rather expensive compared
> to downloading a tarball

You don't have to clone anything.  If all you want is a tarball, use
'git archive --remote'.

> As you say, these tarballs are not available for very long.

*nod*, that's why we switched to a git repo, and extract the tarballs
from it.

Back when we made the switch, I wrote extensive documentation and
recipes as to the possibilities of getting to the GNU Linux-libre
sources from it, how to verify signatures, etc.  It was to be assessed
by then maintainers of Guix recipes to build GNU Linux-libre before
publishing, but it doesn't look like I ever got feedback on them, and
they hvae been unpublished so far.

Is it correct to assume that they have never reached you?  Would you
like to have a look and let me know whether they make sense to you?  I'd
be glad to dig them up and share them with you, if so, and to take your
feedback towards making them available to the public at large.

Thanks,

-- 
Alexandre Oliva, happy hacker                https://FSFLA.org/blogs/lxo/
   Free Software Activist                       GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>

Re: linux-libre source tarballs

[Leo Famulari]

On Mon, May 03, 2021 at 01:39:54PM -0300, Alexandre Oliva wrote:
> You don't have to clone anything.  If all you want is a tarball, use
> 'git archive --remote'.

Thanks, I didn't know about this feature. Is the tarball generation
"stable" across Git releases? Or is it subject to change? We had this
issue with Git archive creation in the past.

> Is it correct to assume that they have never reached you?  Would you
> like to have a look and let me know whether they make sense to you?  I'd
> be glad to dig them up and share them with you, if so, and to take your
> feedback towards making them available to the public at large.

Personally, as a non-maintainer, I'd prefer to preserve the status quo
of our linux-libre packaging. If the maintainers decide that it should
be changed, I'll follow their decision.

Re: linux-libre source tarballs

[Alexandre Oliva]

On May  3, 2021, Leo Famulari <leo@famulari.name> wrote:

> Is the tarball generation "stable" across Git releases? Or is it
> subject to change? We had this issue with Git archive creation in the
> past.

I have heard about early format changes, but since Linux itself has used
git archive to generate tarballs on the fly, and published digital
signatures based on archives generated this way, I presume an
expectation that changes to the format won't be made without a very
compelling reason.

-- 
Alexandre Oliva, happy hacker                https://FSFLA.org/blogs/lxo/
   Free Software Activist                       GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>

Re: linux-libre source tarballs

[Leo Famulari]

On Thu, May 06, 2021 at 01:10:45AM -0300, Alexandre Oliva wrote:
> I have heard about early format changes, but since Linux itself has used
> git archive to generate tarballs on the fly, and published digital
> signatures based on archives generated this way, I presume an
> expectation that changes to the format won't be made without a very
> compelling reason.

I asked because, historically, the linux-libre project has chosen not to
store tarballs of its generated linux-libre source trees.

Linux, on the other hand, does store their tarballs in perpetuity, even
if they are created with `git archive`.

Do you know for sure that Linux generates tarballs "on the fly" and does
not simply preserve the signed tarballs forever?

Otherwise, the PGP signatures could go "stale" at some point, and that
would be unprecedented and unacceptable.

Re: linux-libre source tarballs

[Maxim Cournoyer]

Hello,

Alexandre Oliva <lxoliva@fsfla.org> writes:

[...]

> Back when we made the switch, I wrote extensive documentation and
> recipes as to the possibilities of getting to the GNU Linux-libre
> sources from it, how to verify signatures, etc.  It was to be assessed
> by then maintainers of Guix recipes to build GNU Linux-libre before
> publishing, but it doesn't look like I ever got feedback on them, and
> they hvae been unpublished so far.
>
> Is it correct to assume that they have never reached you?  Would you
> like to have a look and let me know whether they make sense to you?  I'd
> be glad to dig them up and share them with you, if so, and to take your
> feedback towards making them available to the public at large.

I for one have not seen them.  Would you mind making them available
somewhere?  I'd be interested in having a look.

Thanks,

Maxim

Re: linux-libre source tarballs

[Alexandre Oliva]

On May  6, 2021, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> Alexandre Oliva <lxoliva@fsfla.org> writes:

>> recipes as to the possibilities of getting to the GNU Linux-libre
>> sources from it, how to verify signatures, etc

> I for one have not seen them.  Would you mind making them available
> somewhere?  I'd be interested in having a look.

Not as extensive as I thought ;-)
unless I found only an early version,
but so here it goes:


* Download one release as git checkout:

git clone -b sources/v$version --single-branch \
  git://linux-libre.fsfla.org/releases.git linux-libre-$version/


* Example substitutions:

version=5.7-gnu upversion=5.7
prefix=linux-$upversion/


* If you plan on verifying any releases:

git archive --remote=git://linux-libre.fsfla.org/releases.git \
  --format tar master | tar -x -O keyring.gpg | gpg --import -


* If you plan on verifying release $version:

git archive --remote=git://linux-libre.fsfla.org/releases.git \
  --format tar logs/v$version | tar -x &&
if test -f linux-libre-$version.git.tar.sign; then
  prefix=linux-libre-$version/ targit=.git
else
  prefix=linux-$upversion/ targit=
fi


* Download/archive one release as git tarball:

# Use the conditional prefix above to be able to verify.
git archive --remote=git://linux-libre.fsfla.org/releases.git \
  --format tar.gz --prefix=$prefix sources/v$version \
  -o linux-libre-$version$targit.tar.gz


* Verify one release in git repo:

git archive --format tar --prefix=$prefix sources/v$version |
gpg --verify logs/v$version/linux-libre-$version$targit.tar.sign -


* Verify one release in a (gzip-compressed) tarball:

# Must have used the conditional prefix above to verify.
zcat linux-libre-$version$targit.tar.gz |
gpg --verify linux-libre-$version$targit.tar.sign -


* Download and verify a released compressed tarball:

ext=lz comp=lzip # ext=bz2 comp=bzip2 # ext=xz comp=xz
wget https://linux-libre.fsfla.org/pub/linux-libre/releases\
/$version/linux-libre-$version.tar.{$ext,sign} &&
$comp -d < linux-libre-$version.tar.$ext |
gpg --verify linux-libre-$version.tar.sign - # no $targit


-- 
Alexandre Oliva, happy hacker                https://FSFLA.org/blogs/lxo/
   Free Software Activist                       GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>

Re: linux-libre source tarballs

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1896 bytes --]

On 2021-05-01, Leo Famulari wrote:
> On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote:
>> Pragmatically speaking, on slower platforms this is a huge resource
>> overhead. So much so that ci.guix.gnu.org *usually* times out when
>> generating the linux-libre aarch64 tarballs:
>> 
>>   https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic
>
> Thanks for letting me know. I didn't know this was happening.
>
> The immediate solution is for me to make sure the tarballs have built
> before committing the updates. I already do this for x86_64 and I can
> start doing it for aarch64 too.

This has definitely helped sometimes, thanks! I even saw a substitute of
linux-libre for aarch64 earlier today! :)

Still, I'm noticing another problem with the way way the tarballs are
generated on ci.guix.gnu.org ...

When it generates a tarball, all the various packages independently try
to recreate the source tarball; so you have at least fours jobs
("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers",
"linux-libre-bpf") all concurrently trying to build the very same
very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them
might succeed, but the others may not, and even though one of them
succeeded, none of the failing ones retry...

Not knowing exactly how ci.guix.gnu.org works, would it make sense to
create a tarball package instead of the ... computed origin(?) tarball,
so it could be better represented in the package dependency graph, and
the various linux-libre-* packages can wait till it is available rather
than all trying to recreate the same thing?

That still requires the tarball generation to not time out in the first
place, but maybe it would help with the resource limitations a bit to
only build the source tarball once per architecture?


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: linux-libre source tarballs

[Leo Famulari]

On Sat, May 29, 2021 at 09:50:36PM -0700, Vagrant Cascadian wrote:
> When it generates a tarball, all the various packages independently try
> to recreate the source tarball; so you have at least fours jobs
> ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers",
> "linux-libre-bpf") all concurrently trying to build the very same
> very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them
> might succeed, but the others may not, and even though one of them
> succeeded, none of the failing ones retry...
> 
> Not knowing exactly how ci.guix.gnu.org works, would it make sense to
> create a tarball package instead of the ... computed origin(?) tarball,
> so it could be better represented in the package dependency graph, and
> the various linux-libre-* packages can wait till it is available rather
> than all trying to recreate the same thing?

If I understand correctly, this was recently fixed in Cuirass:

https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=d1a95e8b33b454a45bda506a22a8b9d9d2c8b16e

Re: linux-libre source tarballs

[Mathieu Othacehe]

Hello,

>> Not knowing exactly how ci.guix.gnu.org works, would it make sense to
>> create a tarball package instead of the ... computed origin(?) tarball,
>> so it could be better represented in the package dependency graph, and
>> the various linux-libre-* packages can wait till it is available rather
>> than all trying to recreate the same thing?
>
> If I understand correctly, this was recently fixed in Cuirass:
>
> https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=d1a95e8b33b454a45bda506a22a8b9d9d2c8b16e

With the recent Cuirass commits, the notion of build dependencies have
been introduced. If you have a look to this page:
https://ci.guix.gnu.org/build/497096/details, you will see that the
"git" build won't start until all the other builds it depends are
successfully completed (xmlto, subversion, tk ...).

The problem with linux-libre and a few other packages is that some of
the derivations they depend do not correspond to proper packages that
are identified and built by Cuirass. You will see here:
https://ci.guix.gnu.org/build/512266/details that linux-libre has no
build dependencies.

If we could translate the computed origin method into a package, I think
it could solve this issue. I also noticed a similar issue with
ublock-origin and the make-chromium-extension method.

Thanks,

Mathieu

source tarballs potentially built for each derivation

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 3206 bytes --]

Turning this conversation into a bug, original thread around here:

  https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427

On 2021-05-29, Vagrant Cascadian wrote:
> On 2021-05-01, Leo Famulari wrote:
>> On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote:
>>> Pragmatically speaking, on slower platforms this is a huge resource
>>> overhead. So much so that ci.guix.gnu.org *usually* times out when
>>> generating the linux-libre aarch64 tarballs:
>>> 
>>>   https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic
>>
>> Thanks for letting me know. I didn't know this was happening.
>>
>> The immediate solution is for me to make sure the tarballs have built
>> before committing the updates. I already do this for x86_64 and I can
>> start doing it for aarch64 too.
>
> This has definitely helped sometimes, thanks! I even saw a substitute of
> linux-libre for aarch64 earlier today! :)
>
> Still, I'm noticing another problem with the way way the tarballs are
> generated on ci.guix.gnu.org ...
>
> When it generates a tarball, all the various packages independently try
> to recreate the source tarball; so you have at least fours jobs
> ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers",
> "linux-libre-bpf") all concurrently trying to build the very same
> very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them
> might succeed, but the others may not, and even though one of them
> succeeded, none of the failing ones retry...
>
> Not knowing exactly how ci.guix.gnu.org works, would it make sense to
> create a tarball package instead of the ... computed origin(?) tarball,
> so it could be better represented in the package dependency graph, and
> the various linux-libre-* packages can wait till it is available rather
> than all trying to recreate the same thing?
>
> That still requires the tarball generation to not time out in the first
> place, but maybe it would help with the resource limitations a bit to
> only build the source tarball once per architecture?

This seems to still be an issue for ci.guix.gnu.org, but the
linux-libre* substitutes for aarch64 seem to be available on
https://bordeaux.guix.gnu.org ...

$ guix weather linux-libre linux-libre-arm64-generic
computing 2 package derivations for aarch64-linux...
looking for 2 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  0.0% substitutes available (0 out of 2)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.740 seconds per request (0.7 seconds in total)
  1.4 requests per second

  0.0% (0 out of 2) of the missing items are queued
  1 queued builds
      aarch64-linux: 1 (100.0%)
  build rate: .00 builds per hour
      x86_64-linux: 0.00 builds per hour
      aarch64-linux: 0.00 builds per hour
      i686-linux: 0.00 builds per hour
looking for 2 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  100.0% substitutes available (2 out of 2)
  83.9 MiB of nars (compressed)
  202.2 MiB on disk (uncompressed)
  (continuous integration information unavailable)


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: source tarballs potentially built for each derivation

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

On 2021-08-01, Vagrant Cascadian wrote:
> Turning this conversation into a bug, original thread around here:
>
>   https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427

For reference:

  bug#49810: source tarballs potentially built for each derivation


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]