bug#46849: ELPA packages are fetched from unstable url -> not reproducible

bug#46849: ELPA packages are fetched from unstable url -> not reproducible

[Johannes Rosenberger]

Hey Guixers,

i think guix makes the same mistake Nixpkgs make (at least when I looked up 
what guix is doing around two weeks ago):

They fetch the uncompressed tars built by ELPA.

These are only available for the newest version of a package.
ELPA keeps compressed archives only of around 20 hand-selected versions. 
All package versions are kept in their git repo, which is a complete archive,
but there you must somehow extract the commit hash of a version.

Details are here:

- https://github.com/ttuegel/emacs2nix/issues/55
- https://github.com/NixOS/nixpkgs/issues/110796
- https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46441

I proposed possible solutions in the Nixpkgs issue.


Best,

Johannes

bug#46849: ELPA packages are fetched from unstable url -> not reproducible

[zimoun]

Hi,

I did a mistake and the bug report had not been CC.

Cheers,
simon

-------------------- Start of forwarded message --------------------
From: zimoun <zimon.toutoune@gmail.com>
To: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
 reproducible
Date: Fri, 05 Mar 2021 02:32:11 +0100

Hi,

Thanks for the notification.

On Mon, 01 Mar 2021 at 14:15, Johannes Rosenberger <johannes@jorsn.eu> wrote:

> These are only available for the newest version of a package.
> ELPA keeps compressed archives only of around 20 hand-selected versions. 
> All package versions are kept in their git repo, which is a complete archive,
> but there you must somehow extract the commit hash of a version.

So it would break the “guix time-machine”, right?

There is 2 solutions:

 1- trust the future Tarball Heritage [1]
 2- switch to git-fetch all the ELPA packages.

> - https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46441

About #2, I am confused by this quote:

        If you can work from the elpa.git instead, then you'll avoid
        those problems (but the content is slightly different, so it
        might be less convenient).


1: <https://git.ngyro.com/disarchive>


All the best,
simon
-------------------- End of forwarded message --------------------

bug#46849: ELPA packages are fetched from unstable url -> not reproducible

[zimoun]

Follow-up.

-------------------- Start of forwarded message --------------------
Date: Fri, 05 Mar 2021 12:56:27 +0100
From: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
 reproducible
To: zimoun <zimon.toutoune@gmail.com>

Hi Simon,


Excerpts from zimoun's message of March 5, 2021 2:32 am:

> On Mon, 01 Mar 2021 at 14:15, Johannes Rosenberger <johannes@jorsn.eu> wrote:
> 
>> These are only available for the newest version of a package.
>> ELPA keeps compressed archives only of around 20 hand-selected versions. 
>> All package versions are kept in their git repo, which is a complete archive,
>> but there you must somehow extract the commit hash of a version.
> 
> So it would break the “guix time-machine”, right?

Not only this. In Nixpkgs it broke the release of auctex in the 
stable branch, because this wasn't at the newest version.
The old version was still available lz-compressed, but there is no 
guarantee for this.

> There is 2 solutions:
> 
>  1- trust the future Tarball Heritage [1]
>  2- switch to git-fetch all the ELPA packages.

I documented (2) there:

https://github.com/NixOS/nixpkgs/issues/110796#issuecomment-779297144

There is one third solution:

   3- trust archive.org

In Nixpkgs we also add archive.org urls as secondary source urls for 
proprietary printer drivers.

>> - https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46441
> 
> About #2, I am confused by this quote:
> 
>         If you can work from the elpa.git instead, then you'll avoid
>         those problems (but the content is slightly different, so it
>         might be less convenient).

I don't understand this sentence either, because the file

http://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/elpa-admin.el?h=elpa-admin

seems to create the packages, so every package in the elpa built from 
the git should be the same. One could check whether all packages on ELPA 
are also in the git and vice versa. Also, some packages might not be 
`external` in the language of ELPA, so not residing in an `external/*` 
branch.


Best,

Johannes
-------------------- End of forwarded message --------------------

bug#46849: ELPA packages are fetched from unstable url -> not reproducible

[zimoun]

Follow up 2.

-------------------- Start of forwarded message --------------------
Date: Fri, 05 Mar 2021 13:08:05 +0100
From: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
 reproducible
To: zimoun <zimon.toutoune@gmail.com>

Excerpts from Johannes Rosenberger's message of March 5, 2021 12:56 pm:

> Excerpts from zimoun's message of March 5, 2021 2:32 am:
> 
>> There is 2 solutions:
>> 
>>  1- trust the future Tarball Heritage [1]
>>  2- switch to git-fetch all the ELPA packages.
>   3- trust archive.org

and maybe a fourth one:

    4- https://www.softwareheritage.org/
       (Blog entry about Nix & this by Tweag: https://www.softwareheritage.org/)

Best,

Johannes

-------------------- End of forwarded message --------------------

bug#46849: ELPA packages are fetched from unstable url -> not reproducible

[zimoun]

Follow up 3.

-------------------- Start of forwarded message --------------------
From: zimoun <zimon.toutoune@gmail.com>
To: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
 reproducible
Date: Fri, 05 Mar 2021 13:31:09 +0100

Hi Johannes,

On Fri, 05 Mar 2021 at 13:08, Johannes Rosenberger <johannes@jorsn.eu> wrote:

>>> There is 2 solutions:
>>>
>>>  1- trust the future Tarball Heritage [1]
>>>  2- switch to git-fetch all the ELPA packages.
>>   3- trust archive.org

About archive.org, I do not know.  Currently, there is no fallback in
Guix to it that I am aware, and nothing planned AFAIK.

> and maybe a fourth one:
>
>     4- https://www.softwareheritage.org/
>        (Blog entry about Nix & this by Tweag: https://www.softwareheritage.org/)

Yeah, this is what I called #1. :-) Currently, via the ’nixguix’ SWH
loader [1], packages using url-fetch are archived via the file [2].
However, work remains to have a full robust end-to-end solution:

  a) not all the extensions of ’url-fetch’ are archived (and I do not
remember the status about the .el)

  b) the fallback is not robust because of inconsistent addresses
between SWH (swh-id) and the-rest-of-the-world (checksum hashes)–to say
it quickly.

The aim of the disarchive’s project [3] is to address b) by creating a
bridge, i.e., stores in a separate database [4] the structure of the
metadata and then rebuild the archive from a checksum using the files
addressed by swh-id.


1: <https://docs.softwareheritage.org/devel/_modules/swh/loader/package/nixguix.html>
2: <http://guix.gnu.org/sources.json>
3: <https://git.ngyro.com/disarchive>
4: <https://git.ngyro.com/disarchive-db/>


Cheers,
simon
-------------------- End of forwarded message --------------------