disarchive: disassemble then assemble tarball

disarchive: disassemble then assemble tarball

[Simon Tournier]

Hi,

Discussing in #62153 [1], it is noticed that Disarchive is not working
as expected.

For instance, using the ’gnu’ folder containing ’.go’ files.

    $ tar cvf my-gnu.tar gnu

    $ guix shell disarchive guile guile-lzma
    $ mkdir sha256

    $ guile
 scheme@(guile-user)> ,module(disarchive assemblers tarball)
 scheme@(disarchive assemblers tarball)> (assemble-tarball (disassemble-tarball "my-gnu.tar") ".")
 scheme@(disarchive assemblers tarball)> ,q

    $ ls sha256/
7a94f577d3235a5ab954f0bc56b15fbe5518c56354c4235574618f6e666f6a38
    $ file sha256/7a94f577d3235a5ab954f0bc56b15fbe5518c56354c4235574618f6e666f6a38
sha256/7a94f577d3235a5ab954f0bc56b15fbe5518c56354c4235574618f6e666f6a38: POSIX tar archive (GNU)

    $ tar tf sha256/7a94f577d3235a5ab954f0bc56b15fbe5518c56354c4235574618f6e666f6a38
gnu/
gnu/ci.scm
gnu/tests/databases.go
gnu/packages/gpodder.scm

[...]

gnu/build/accounts.go
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

    $ disarchive disassemble my-gnu.tar > spec
Disassembling the tarball my-gnu.tar
Read 2741 headers
Extracting the tarball my-gnu.tar
Disassembling the directory my-gnu
Saving directory in cache: /tmp/disarchive-directory.GGqLse/sha256/e9013143a529feb6568dbe42d6587d6fcebe4b264f1e955cc0d614c52125c998
Finished disassembly of my-gnu.tar
Checking that it can be assembled... ok
Checking that it can be deserialized... ok

    $ tail spec
       (chksum 5343)))
    (padding 0)
    (input (directory-ref
             (version 0)
             (name "my-gnu")
             (addresses
               (swhid "swh:1:dir:5acd725958b0642e98cabb4cb2db670f2e32384e"))
             (digest
               (sha256
                 "e9013143a529feb6568dbe42d6587d6fcebe4b264f1e955cc0d614c52125c998"))))))

Well, I am not sure to understand why the name “my-gnu”.  And then I do
not understand how to use “disarchive assemble”; I get something like:

--8<---------------cut here---------------start------------->8---
Checking my-gnu digest... fail
Output is incorrect
--8<---------------cut here---------------end--------------->8---

What do I do wrong?

1: https://issues.guix.gnu.org/msgid/87h6ugapwh.fsf@gmail.com


Cheers,
simon

Re: disarchive: disassemble then assemble tarball

[Timothy Sample]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Discussing in #62153 [1], it is noticed that Disarchive is not working
> as expected.

To be fair, Oleg is experimenting with reusing some internal Disarchive
code in another context.  That experiment didn’t go well, and Oleg came
to a good (IMO) conclusion: try the tarball code from Gash-Utils.  (The
longer story is that the tarball code in Disarchive is nicer, but it is
specialized for the Disarchive use case.  Sometime I might de-specialize
it and use it in Gash-Utils, but Gash-Utils has difficult constraints
due to its use during bootstrapping.  We’ll see.)

>  scheme@(guile-user)> ,module(disarchive assemblers tarball)
>  scheme@(disarchive assemblers tarball)> (assemble-tarball (disassemble-tarball "my-gnu.tar") ".")
>
> [...]
>
>     $ tar tf sha256/7a94f577d3235a5ab954f0bc56b15fbe5518c56354c4235574618f6e666f6a38
>
> [...]
>
> gnu/build/accounts.go
> tar: Unexpected EOF in archive
> tar: Error is not recoverable: exiting now

The ‘assemble-tarball’ procedure assumes (without checking) that it has
access to a copy of its input in the “workspace” directory.  Normally,
it would be called from the generic ‘assemble’ procedure, which ensures
this (and also checks the output).  It probably just wrote all the
tarball headers without the file contents, producing a corrupt tarball.
Indeed, if you set up logging (using ‘%disarchive-log-port’) it prints a
bunch of “Ignoring irregular file” warnings.

> Well, I am not sure to understand why the name “my-gnu”.

It’s just ‘(basename "my-gnu.tar" ".tar")’.  The names are designed to
aid a human while debugging.  To my knowledge, Disarchive doesn’t use
them for anything.  You can use set the name by passing “--name” to the
‘disassemble’ subcommand.

> And then I do not understand how to use “disarchive assemble”; I get
> something like:
>
> Checking my-gnu digest... fail
> Output is incorrect
>
> What do I do wrong?

You need to put “gnu” in a directory by itself.  First,

  $ mkdir base

Then either

  $ mv gnu base/

or

  $ tar -C base -xf my-gnu.tar

Then you can run

  $ disarchive assemble base spec

In other words, it’s your old nemesis:

  https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00063.html

:)  (I tease only because it’s confused me many times too!)


-- Tim

Re: disarchive: disassemble then assemble tarball

[Simon Tournier]

Hi,

On Mon, 03 Apr 2023 at 14:19, Timothy Sample <samplet@ngyro.com> wrote:

> The ‘assemble-tarball’ procedure assumes (without checking) that it has
> access to a copy of its input in the “workspace” directory.  Normally,
> it would be called from the generic ‘assemble’ procedure, which ensures
> this (and also checks the output).  It probably just wrote all the
> tarball headers without the file contents, producing a corrupt tarball.
> Indeed, if you set up logging (using ‘%disarchive-log-port’) it prints a
> bunch of “Ignoring irregular file” warnings.

Ok, but I was expecting a kind of identity when composing
’disassemble-tarball’ and ’assemble-tarball’.  And even, that the
identity properties of assembling some disassembling would be some
tests. :-)


>> Well, I am not sure to understand why the name “my-gnu”.
>
> It’s just ‘(basename "my-gnu.tar" ".tar")’.  The names are designed to
> aid a human while debugging.  To my knowledge, Disarchive doesn’t use
> them for anything.  You can use set the name by passing “--name” to the
> ‘disassemble’ subcommand.

Thanks for explaining.


> In other words, it’s your old nemesis:
>
>   https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00063.html
>
> :)  (I tease only because it’s confused me many times too!)

Arf, probably because the behaviour appears to me unexpected. :-)

Well, maybe some convention with trailing / would be helpful, as with
’rsync’.


Cheers,
simon