Disarchive database includes tar.xz archives

Disarchive database includes tar.xz archives

[Ludovic Courtès]

Hello Guix!

The database at https://disarchive.guix.gnu.org now includes metadata
for tar.xz archives, like these:

  https://disarchive.guix.gnu.org/sha256/e831b3a86091496cdba720411f9748de81507798f6130adeaef872d206e1b057
  https://disarchive.guix.gnu.org/sha256/daa7060a56f2804e9b74c8d7e7fe8beed12b43aab2789a38585183fcc17b8a13

Computing these has been possible since Disarchive 0.4.0, which Timothy
released in Dec. 2021 but we had not updated
‘etc/disarchive-manifest.scm’ to disassemble tar.xz archives in addition
to tar.gz.

For the record, the database is computed at
<https://ci.guix.gnu.org/jobset/disarchive>.  With this change, the
database goes from 123 MiB to 234 MiB (with individual files gzipped).

Note commit eab5366e679c149cf0a5abeb9a196700b171d9da excludes that
Chromium tarball: it takes 1.4 GiB (!) and when running ‘disarchive
disassemble’ on it, we hit the default max-silent time of 1h (and (gnu
ci) doesn’t let us specify a different timeout in this particular case.)
We could work around it by making Disarchive print a progress report,
which would be nice to have anyway.

Ludo’.

Re: Disarchive database includes tar.xz archives

[zimoun]

Hi,

Cool!

On dim., 07 août 2022 at 19:27, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> Computing these has been possible since Disarchive 0.4.0, which Timothy
> released in Dec. 2021 but we had not updated
> ‘etc/disarchive-manifest.scm’ to disassemble tar.xz archives in addition
> to tar.gz.

I do not the remember the status of tar.xz in nixguix SWH loader.


> Note commit eab5366e679c149cf0a5abeb9a196700b171d9da excludes that
> Chromium tarball: it takes 1.4 GiB (!) and when running ‘disarchive
> disassemble’ on it, we hit the default max-silent time of 1h (and (gnu
> ci) doesn’t let us specify a different timeout in this particular case.)
> We could work around it by making Disarchive print a progress report,
> which would be nice to have anyway.

Or we could also ask if we need to disassemble and archive this Chromium
tarball.  Does the world need to hold such big program for just web
browsing ?  I am half-joking. :-)


Cheers,
simon

Re: Disarchive database includes tar.xz archives

[Ludovic Courtès]

Hello!

(+Cc: swh-devel)

zimoun <zimon.toutoune@gmail.com> skribis:

> I do not the remember the status of tar.xz in nixguix SWH loader.

Looking at
<https://forge.softwareheritage.org/source/swh-core/browse/master/swh/core/tarball.py$239>,
it would seem that tar.xz is not supported (but tar.bz2 and tar.zst
are).

Am I right?

That said, I ran ‘guix lint -c archival’ on several tar.xz packages and
couldn’t find any whose source is missing from SWH.

Thanks,
Ludo’.

Re: [swh-devel] Disarchive database includes tar.xz archives

[Franck Bret]

Hi,

For what I can say tar.xz support is not explicit in tarball.py but I've 
managed tar.xz archive for Arch Linux loader and it worked.

See loader/package/arch/tests/test_arch.py

Le 8/29/22 à 11:43, Ludovic Courtès a écrit :
> Hello!
>
> (+Cc: swh-devel)
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> I do not the remember the status of tar.xz in nixguix SWH loader.
> Looking at
> <https://forge.softwareheritage.org/source/swh-core/browse/master/swh/core/tarball.py$239>,
> it would seem that tar.xz is not supported (but tar.bz2 and tar.zst
> are).
>
> Am I right?
>
> That said, I ran ‘guix lint -c archival’ on several tar.xz packages and
> couldn’t find any whose source is missing from SWH.
>
> Thanks,
> Ludo’.

Re: [swh-devel] Disarchive database includes tar.xz archives

[Valentin Lorentz]

Hi,

ADDITIONAL_ARCHIVE_FORMATS complements formats already supported 
natively by Python's shutil module (listed here: 
https://docs.python.org/3/library/shutil.html#shutil.unpack_archive ) 
which includes tar.bz2

On 29/08/2022 11:43, Ludovic Courtès wrote:
> Hello!
> 
> (+Cc: swh-devel)
> 
> zimoun <zimon.toutoune@gmail.com> skribis:
> 
>> I do not the remember the status of tar.xz in nixguix SWH loader.
> 
> Looking at
> <https://forge.softwareheritage.org/source/swh-core/browse/master/swh/core/tarball.py$239>,
> it would seem that tar.xz is not supported (but tar.bz2 and tar.zst
> are).
> 
> Am I right?
> 
> That said, I ran ‘guix lint -c archival’ on several tar.xz packages and
> couldn’t find any whose source is missing from SWH.
> 
> Thanks,
> Ludo’.

Re: [swh-devel] Disarchive database includes tar.xz archives

[Ludovic Courtès]

Hi,

Valentin Lorentz <valentin.lorentz@inria.fr> skribis:

> ADDITIONAL_ARCHIVE_FORMATS complements formats already supported
> natively by Python's shutil module (listed here:
> https://docs.python.org/3/library/shutil.html#shutil.unpack_archive )
> which includes tar.bz2

Perfect.  Thanks for explaining!

Ludo’.