Software Heritage API

Software Heritage API

[Ludovic Courtès]

Hello Guix!

I couldn’t attend the Software Heritage talk at FOSDEM, but they had
good news:

  https://www.softwareheritage.org/2017/02/04/archive-api/

The API is currently limited to meta-data (retrieving the actual data
returns 401.)  Still that looks pretty cool already.

Ludo’.

Re: Software Heritage API

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

On Mon, Feb 06, 2017 at 10:34:17AM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> I couldn’t attend the Software Heritage talk at FOSDEM, but they had
> good news:
> 
>   https://www.softwareheritage.org/2017/02/04/archive-api/
> 
> The API is currently limited to meta-data (retrieving the actual data
> returns 401.)  Still that looks pretty cool already.
> 
> Ludo’.
> 

I've kept this tagged to take a look at it later. I checked the sha1sum
of swig-3.0.10.tar.gz and it gave me a valid URL.
https://archive.softwareheritage.org/api/1/content/c672b8535394cfb204c70de7c66e69fb20a95647/
https://archive.softwareheritage.org/api/1/content/sha1:c672b8535394cfb204c70de7c66e69fb20a95647/
https://archive.softwareheritage.org/api/1/content/sha256:2939aae39dec06095462f1b95ce1c958ac80d07b926e48871046d17c0094f44c/
If you take a look at the page(s), '/raw' can only be appended to the
sha1 (or blank) URLs to download the source, which currently returns
401.

Currently our "magic mirrors" search hydra based on the hash; in order
to check here also for the source we'd have to undo the base32 hash, and
then either transform the sha256 hash to a sha1 hash, or use two API
calls, the first to check for the source and the second to get and use
the url to download it. A quick check online makes me think it's not
possible to take a sha256 hash and get the sha1 hash of that file.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Software Heritage API

[Ludovic Courtès]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> skribis:

> I've kept this tagged to take a look at it later. I checked the sha1sum
> of swig-3.0.10.tar.gz and it gave me a valid URL.
> https://archive.softwareheritage.org/api/1/content/c672b8535394cfb204c70de7c66e69fb20a95647/
> https://archive.softwareheritage.org/api/1/content/sha1:c672b8535394cfb204c70de7c66e69fb20a95647/
> https://archive.softwareheritage.org/api/1/content/sha256:2939aae39dec06095462f1b95ce1c958ac80d07b926e48871046d17c0094f44c/
> If you take a look at the page(s), '/raw' can only be appended to the
> sha1 (or blank) URLs to download the source, which currently returns
> 401.

Be aware that Software Heritage (SWH) stores only raw commits and not
tarballs (or not yet).  That means that you may be able to find the
“3.0.10” tag of SWIG, but not swig-3.0.10.tar.gz.  See:

  https://sympa.inria.fr/sympa/arc/swh-devel/2016-09/msg00000.html

> Currently our "magic mirrors" search hydra based on the hash; in order
> to check here also for the source we'd have to undo the base32 hash, and
> then either transform the sha256 hash to a sha1 hash, or use two API
> calls, the first to check for the source and the second to get and use
> the url to download it. A quick check online makes me think it's not
> possible to take a sha256 hash and get the sha1 hash of that file.

As it is now, SWH could only help us for Git checkouts, not for
tarballs.

There is no way to “convert” a SHA256 hash to SHA1 or similar, though,
but apparently SWH supports SHA256 anyway.

The second problem, though, is that the way we compute the hash of a
directory differs from the way they do:

  https://sympa.inria.fr/sympa/arc/swh-devel/2016-07/msg00018.html

Essentially, Guix computes the hash of the nar (“normalized archive”) of
the directory, whereas SWH computes a hash over the Git tree
representation.

AFAICS this cannot be overcome without manually specifying the
git-tree-hash in our ‘origin’ objects.

Ludo’.