How to compute SWHID? (with Guix/Disarchive)

How to compute SWHID? (with Guix/Disarchive)

[zimoun]

Hi,

Giving a look at Disarchive, I found how to compute Git-based
serialization hash and somehow serialization methods of "guix hash"
needs some clearning; considering '--recursive' is 'nar' serialization
which is a better name.  Anyway, see [1]. :-)

I would like to add SWH-based serialization hash but I do not find if
a function already does the hard work.  Any pointer?

Moreover, I would like to add* a new export format to "guix show"
using BibTeX format proposed by SWH.  It would help when writing
paper. ;-)

*add: discussed long time ago but because I was not writing a paper
using tools deployed by Guix, I was not bitten enough to complete. ;-)

1: <https://issues.guix.gnu.org/51307#17>


Cheers,
simon

Re: How to compute SWHID? (with Guix/Disarchive)

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Giving a look at Disarchive, I found how to compute Git-based
> serialization hash and somehow serialization methods of "guix hash"
> needs some clearning; considering '--recursive' is 'nar' serialization
> which is a better name.  Anyway, see [1]. :-)

Neat!

> I would like to add SWH-based serialization hash but I do not find if
> a function already does the hard work.  Any pointer?

I think it’s ‘git-hash-directory’ in (disarchive git-hash).

The other day I learned that the Git CLI ignores empty directories, but
the Git format itself has nothing against empty directories.  Thus SWH
serializes in exactly the same way as Git.

(Can you confirm, Timothy?)

> Moreover, I would like to add* a new export format to "guix show"
> using BibTeX format proposed by SWH.  It would help when writing
> paper. ;-)

That’d be fun!  Maybe we need a separate tool set for scientific
authoring?

Thanks,
Ludo’.

Re: How to compute SWHID? (with Guix/Disarchive)

[Timothy Sample]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> Giving a look at Disarchive, I found how to compute Git-based
>> serialization hash and somehow serialization methods of "guix hash"
>> needs some clearning; considering '--recursive' is 'nar' serialization
>> which is a better name.  Anyway, see [1]. :-)
>
> Neat!
>
>> I would like to add SWH-based serialization hash but I do not find if
>> a function already does the hard work.  Any pointer?
>
> I think it’s ‘git-hash-directory’ in (disarchive git-hash).

That’s the one.  I only know what SWH does for a few cases:

  • directory: Use their version of ‘git-hash-directory’.

  • file: Use their version of ‘git-hash-file’ (resulting in a ID like
    “swh:1:cnt:...”).  I don’t know if they ingest regular files like
    this, but if they ingested the file through another means, it will
    have that ID.

  • git: Read the directory ID from the Git database.  This is
    essentially ‘git rev-parse HEAD:’, where the colon at the end tells
    Git to get the “tree” (directory) ID rather than the commit ID.
    (I’m not sure if guile-git supports this; so far I’ve just been
    shelling out to Git.)

  • hg: Use their version of ‘git-hash-directory’ excluding the “.hg”
    directory.

In my work, I’ve been strict about keeping the Git directory IDs based
on the Git database (“.git”) rather than computing them using
‘git-hash-directory’.  Since Guix deletes the Git database before
putting a checkout in the store, that option may not be available to you
(unless you download the repository again).  I’m not sure how much of
problem this would be in practice.  There may be a few edge cases with
submodules and “.gitattributes” to watch out for.

My guess is that as it stands, if a repo has a “.gitattributes” file,
running ‘git-hash-directory’ on the checkout will produce a directory ID
that SWH doesn’t have (they will ignore it, but we will include it).  A
corollary of this guess is that our SWH fallback code for Git will fail
for a repo that has a “.gitattributes” file, since we include it in the
nar hash, but SWH will not provide it.  (I say “guess” because this is
based on some stuff I observed when writing that procedure several
months ago – I haven’t verified any of this.  See also
<https://issues.guix.gnu.org/48540>, which is the same problem but with
submodules instead of “.gitattributes”).

Sorry but all I have to offer is doom and gloom on this one.  :(  You
might be able to get ‘git-hash-directory’ to work well enough on the Git
checkouts that Guix puts in the store, but you’ll have to be careful!

> The other day I learned that the Git CLI ignores empty directories, but
> the Git format itself has nothing against empty directories.  Thus SWH
> serializes in exactly the same way as Git.
>
> (Can you confirm, Timothy?)

I can confirm that a Git tree node of the form

    40000 empty-directory 4b825dc642cb6eb9a060e54bf8d69288fbee4904

theoretically represents an empty directory named “empty-directory”.
The hash is computed like this:

    $ printf 'tree 0\0' | sha1sum
    4b825dc642cb6eb9a060e54bf8d69288fbee4904  -

I don’t know anything about where Git excludes this or what would happen
if you manually constructed a Git repo with empty directories, though!


-- Tim

guix show --bibtex (was: How to compute SWHID? (with Guix/Disarchive))

[Liliana Marie Prikler]

Am Mittwoch, den 01.12.2021, 11:12 +0100 schrieb zimoun:
> Moreover, I would like to add* a new export format to "guix show"
> using BibTeX format proposed by SWH.  It would help when writing
> paper. ;-)
> 
> *add: discussed long time ago but because I was not writing a paper
> using tools deployed by Guix, I was not bitten enough to complete. 
> ;-)
Given that `guix show' output is well-structured (it uses the Recutils
format), couldn't you post-process to form a bibtex file?  In
particular, you could use recfmt IIUC.

That said, there would be stuff like author etc. missing from Guix'
metadata.  What kind of output were you hoping to produce?

Cheers

Re: guix show --bibtex (was: How to compute SWHID? (with Guix/Disarchive))

[zimoun]

Hi,

On Tue, 7 Dec 2021 at 18:07, Liliana Marie Prikler
<liliana.prikler@gmail.com> wrote:

> Given that `guix show' output is well-structured (it uses the Recutils
> format), couldn't you post-process to form a bibtex file?  In
> particular, you could use recfmt IIUC.

Good idea.  For an trivial example,

--8<---------------cut here---------------start------------->8---
$ cat bibtex.tmpl
@example{ {{name}}@{{version}},
title      = {{name}},
url        = {{home-page}},
license    = {{license}},
abstract   = {{synopsis}},
swhid      = {swh:1:...},
repository = "WE HAVE THAT INFO",
author     = "?????",
date       = "?????",
}

$ guix show hello | recfmt -f bibtex.tmpl
@example{ hello@2.10,
title      = hello,
url        = 0.000000,
license    = GPL 3+,
abstract   = Hello, GNU world: An example GNU package  ,
swhid      = {swh:1:...},
repository = "WE HAVE THAT INFO",
author     = "?????",
date       = "?????",
}
--8<---------------cut here---------------end--------------->8---

However, 2 minor annoying comments:

 1. SWHID is not computed by "guix show".
 2. Typing the path to the template; aside recutils is not installed
by default, IIRC.

Well, #2 could be solved with something under ~/.config/guix/current/bin.

About #1, it requires some work and I am not convinced I would like
that Guix does it each time I call "guix show".

> That said, there would be stuff like author etc. missing from Guix'
> metadata.  What kind of output were you hoping to produce?

As much fields as possible.  As the example above more or less.  Well,
use bibtex-software as decribe here:

https://mirrors.chevalier.io/CTAN/macros/latex/contrib/biblatex-contrib/biblatex-software/software-biblatex.pdf


Cheers,
simon

Re: guix show --bibtex (was: How to compute SWHID? (with Guix/Disarchive))

[Liliana Marie Prikler]

Hi,

Am Dienstag, den 07.12.2021, 18:39 +0100 schrieb zimoun:
> --8<---------------cut here---------------start------------->8---
> $ cat bibtex.tmpl
> @example{ {{name}}@{{version}},
> title      = {{name}},
> url        = {{home-page}},
You typo'd on homepage here, the dash is superfluous.
> license    = {{license}},
> abstract   = {{synopsis}},
> swhid      = {swh:1:...},
> repository = "WE HAVE THAT INFO",
I too don't like that origin information is missing from `guix show'. 
Can we fix that somehow?
> author     = "?????",
We don't capture that in Guix, you'd have to look elsewhere IIUC.
> date       = "?????",
January 1st, 1970, what else?
> }
> 
> [...]
> --8<---------------cut here---------------end--------------->8---
I think there are a few things that'd need fixing in recutils itself.

1a. There needs to be a way of escaping {} in templates -- or at least
I haven't found one.
1b. A rec2bib command would be nice, as would be bib2rec.
2. Looking at the output, a rec2rec command would also be nice, i.e.
one that could rename fields.  Though that one would probably be easier
to implement in terms of recfmt.

>  1. SWHID is not computed by "guix show".
Is that a Guix concern, though?  We also don't have author and date, so
you'd have to shell out either way.

>  2. recutils is not installed by default, IIRC.
I don't think that's too hard if you use guix shell (or environment)
inside your scientific workflow -- just add recutils to emacs-citar and
whatever else you only need for scientific writing :)

> As much fields as possible.  As the example above more or
> less.  Well, use bibtex-software as decribe here:
> 
> https://mirrors.chevalier.io/CTAN/macros/latex/contrib/biblatex-contrib/biblatex-software/software-biblatex.pdf
Uhm, what exactly should I extract from this manual?

Cheers,
Liliana

Re: guix show --bibtex (was: How to compute SWHID? (with Guix/Disarchive))

[zimoun]

Hi,


On Tue, 7 Dec 2021 at 19:20, Liliana Marie Prikler
<liliana.prikler@gmail.com> wrote:
> Am Dienstag, den 07.12.2021, 18:39 +0100 schrieb zimoun:

> > --8<---------------cut here---------------start------------->8---
> > $ cat bibtex.tmpl
> > @example{ {{name}}@{{version}},
> > title      = {{name}},
> > url        = {{home-page}},
> You typo'd on homepage here, the dash is superfluous.

Yes, thanks! :-)

> > license    = {{license}},
> > abstract   = {{synopsis}},
> > swhid      = {swh:1:...},
> > repository = "WE HAVE THAT INFO",
> I too don't like that origin information is missing from `guix show'.
> Can we fix that somehow?

I think it is an easy fix but one could raise it violates FSDG.  I do
not want to open the case because from my understanding, display this
info by "guix show" is *strictly* equivalent to have an option
returning true upstream, instead of Guix modified one; discussed at
length. :-)

> > author     = "?????",
> We don't capture that in Guix, you'd have to look elsewhere IIUC.

Yep.  The meaning of that ???? was: please, do not forget to complete
before include that to your paper. :-)

> > date       = "?????",
> January 1st, 1970, what else?

The date of publication.  Guix do not capture this information either.


> I think there are a few things that'd need fixing in recutils itself.

I agree.


> 1a. There needs to be a way of escaping {} in templates -- or at least
> I haven't found one.

Me neither.


> >  1. SWHID is not computed by "guix show".

> Is that a Guix concern, though?  We also don't have author and date, so
> you'd have to shell out either way.

Yes.  It is my main motivation for such feature.

 a) Have something almost compliant with software-bibtex, which
requires swh:1:...
 b) Extract also an entry providing somehow "guix describe" as software-bibtex

The point a) is the key point, IMHO.  Otherwise, a trivial template
(or any half-baked manual converter fits the bill; I previously did
that in another life. :-))

Well, that's why, I agree with Ludo and this feature should be
external to Guix.  For instance, an extension available with "guix
install", as GWL.

> >  2. recutils is not installed by default, IIRC.
> I don't think that's too hard if you use guix shell (or environment)
> inside your scientific workflow -- just add recutils to emacs-citar and
> whatever else you only need for scientific writing :)

Yeah for sure.  I would prefer to have an extension, say
"guix-publishing", available as a regular package, then "guix
publishing <options>" would allow to export, check SWH, and other
stuff I often do when preparing a publication.


> > As much fields as possible.  As the example above more or
> > less.  Well, use bibtex-software as decribe here:
> >
> > https://mirrors.chevalier.io/CTAN/macros/latex/contrib/biblatex-contrib/biblatex-software/software-biblatex.pdf
> Uhm, what exactly should I extract from this manual?

Well, I just wanted to point what I have in mind, for the interested
reader not knowing bibtex-software.


Cheers,
simon