bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Mauricio Collares]

Currently, there's no way to get a permanent link to a package version
that happens to be the current one. For example, auctex is currently at
version 13.0.4; today I can download it from
https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
working as soon as a new version of auctex is released and the old
version gets compressed. This makes it slightly annoying to pin a
particular version of an ELPA package by URL.

I would like to ask the GNU ELPA maintainers to host both
https://elpa.gnu.org/packages/PACKAGE-CURRENTVERSION.lz (permanent) as
well as https://elpa.gnu.org/packagesa/PACKAGE-CURRENTVERSION
(temporary). This would make it easier for people to build reproducible
environments by pinning a package version. I don't particularly care
about exposing it as a link on the website; for my purposes, it's enough
that the file exists. This is perhaps a little bit wasteful, but
compressed versions shouldn't be too big.

Best,
Mauricio

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Basil L. Contovounesios]

[CCing the GNU ELPA maintainer.]

Mauricio Collares <mauricio@collares.org> writes:

> Currently, there's no way to get a permanent link to a package version
> that happens to be the current one. For example, auctex is currently at
> version 13.0.4; today I can download it from
> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
> working as soon as a new version of auctex is released and the old
> version gets compressed. This makes it slightly annoying to pin a
> particular version of an ELPA package by URL.
>
> I would like to ask the GNU ELPA maintainers to host both
> https://elpa.gnu.org/packages/PACKAGE-CURRENTVERSION.lz (permanent) as
> well as https://elpa.gnu.org/packagesa/PACKAGE-CURRENTVERSION
> (temporary). This would make it easier for people to build reproducible
> environments by pinning a package version. I don't particularly care
> about exposing it as a link on the website; for my purposes, it's enough
> that the file exists. This is perhaps a little bit wasteful, but
> compressed versions shouldn't be too big.

Wouldn't it be wasteless if the "current version" URL was symbolic and
resolved to the concrete versioned release?

Thanks,

-- 
Basil

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Stefan Monnier]

>> Currently, there's no way to get a permanent link to a package version
>> that happens to be the current one.

FWIW, the situation is even worse because there's no way to get
a permanent link at all: while we do keep some old versions, we don't
keep them all.  Currently, the limit is set at 20, meaning that we keep
up to 20 old versions around per package, and once we hit that limit we
start pruning the old versions according to a heuristic that intends to
guess which versions are more important to keep.

So that makes me question your "upstream" need.

>> For example, auctex is currently at
>> version 13.0.4; today I can download it from
>> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
>> working as soon as a new version of auctex is released and the old
>> version gets compressed. This makes it slightly annoying to pin a
>> particular version of an ELPA package by URL.

Could you expand a bit on why you need to keep references to old
versions, and why you decided to use URLs for that?
That will hopefully help us see what is the best course of action
from here.


        Stefan "not opposed to building the .lz eagerly"

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Mauricio Collares]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Currently, there's no way to get a permanent link to a package version
>>> that happens to be the current one.
>
> FWIW, the situation is even worse because there's no way to get
> a permanent link at all: while we do keep some old versions, we don't
> keep them all.  Currently, the limit is set at 20, meaning that we keep
> up to 20 old versions around per package, and once we hit that limit we
> start pruning the old versions according to a heuristic that intends to
> guess which versions are more important to keep.
>
> So that makes me question your "upstream" need.
>
>>> For example, auctex is currently at
>>> version 13.0.4; today I can download it from
>>> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
>>> working as soon as a new version of auctex is released and the old
>>> version gets compressed. This makes it slightly annoying to pin a
>>> particular version of an ELPA package by URL.
>
> Could you expand a bit on why you need to keep references to old
> versions, and why you decided to use URLs for that?
> That will hopefully help us see what is the best course of action
> from here.

Hi Stefan,

Thanks for the reply! The particular use case here is to have a
reproducible environment via the Nix package manager. Omitting
irrelevant details, Nix is a source-based package manager whose
"recipes" for building packages typically start by doing the equivalent
of "download this file with this sha256 from this URL" (actually, this
is typically done by fetching a specific commit from a repository, but
in this case I believe tarballs are the only option); that is, all
packages are pinned. This is done for emacs packages too, so it is
possible to have the same exact emacs setup on several machines or to
recover the exact same state if you have to reinstall everything from
scratch for some reason.

No one that uses Nix expects things to work "forevermore" since source
tarballs frequently disappear (people are working on that, though [0]),
but frequently-updated ELPA packages basically "break" reproducibility
every week. More concretely, if I commit my Nix configuration to a Git
repo and a co-worker wants to use it a week from now, it's likely that
the auctex fetch step will fail. Since NixOS releases happen every six
months, it's fair to say a "year's worth" of stability would reduce
user-facing problems to almost zero -- and I guess archiving 20 releases
provides that with room to spare.

Best,
Mauricio

[0] https://www.tweag.io/blog/2020-06-18-software-heritage/

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Mauricio Collares]

Basil L. Contovounesios <contovob@tcd.ie> writes:

> [CCing the GNU ELPA maintainer.]
>
> Mauricio Collares <mauricio@collares.org> writes:
>
>> Currently, there's no way to get a permanent link to a package version
>> that happens to be the current one. For example, auctex is currently at
>> version 13.0.4; today I can download it from
>> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
>> working as soon as a new version of auctex is released and the old
>> version gets compressed. This makes it slightly annoying to pin a
>> particular version of an ELPA package by URL.
>>
>> I would like to ask the GNU ELPA maintainers to host both
>> https://elpa.gnu.org/packages/PACKAGE-CURRENTVERSION.lz (permanent) as
>> well as https://elpa.gnu.org/packagesa/PACKAGE-CURRENTVERSION
>> (temporary). This would make it easier for people to build reproducible
>> environments by pinning a package version. I don't particularly care
>> about exposing it as a link on the website; for my purposes, it's enough
>> that the file exists. This is perhaps a little bit wasteful, but
>> compressed versions shouldn't be too big.
>
> Wouldn't it be wasteless if the "current version" URL was symbolic and
> resolved to the concrete versioned release?


Hi Basil,

Thanks for taking the time to reply! I think my use of "CURRENTVERSION"
in the previous email URL was ambiguous. Concretely, what I want to have
is to be able to fetch auctex 13.0.4 (which, as of 2021-02-12 is the
current version) through a URL that will still serve the exact same file
(auctex 13.0.4, and ideally same sha256) a year from now. If I
understand correctly, your suggestion gives a link that always points to
the newest version, due to ambiguity in my previous email. I apologize
for the confusion.

Best,
Mauricio

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Basil L. Contovounesios]

Mauricio Collares <mauricio@collares.org> writes:

> Thanks for the reply! The particular use case here is to have a
> reproducible environment via the Nix package manager. Omitting
> irrelevant details, Nix is a source-based package manager whose
> "recipes" for building packages typically start by doing the equivalent
> of "download this file with this sha256 from this URL" (actually, this
> is typically done by fetching a specific commit from a repository, but
> in this case I believe tarballs are the only option);

What are the obstacles to fetching from elpa.git here?
E.g. http://git.savannah.gnu.org/cgit/emacs/elpa.git/?h=externals/auctex

Thanks,

-- 
Basil

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Mauricio Collares]

Basil L. Contovounesios <contovob@tcd.ie> writes:

> Mauricio Collares <mauricio@collares.org> writes:
>
>> Thanks for the reply! The particular use case here is to have a
>> reproducible environment via the Nix package manager. Omitting
>> irrelevant details, Nix is a source-based package manager whose
>> "recipes" for building packages typically start by doing the equivalent
>> of "download this file with this sha256 from this URL" (actually, this
>> is typically done by fetching a specific commit from a repository, but
>> in this case I believe tarballs are the only option);
>
> What are the obstacles to fetching from elpa.git here?
> E.g. http://git.savannah.gnu.org/cgit/emacs/elpa.git/?h=externals/auctex

Hi Basil,

If it's possible to do this for any package, then this is a great
alternative! I see there's an elpa-packages file in the ELPA Git repo
pointing to the sources for each package, which is definitely a format
that Nix can work with. A few questions, just to be sure:

1) When does a commit to the package's repo generate a new release on
ELPA? (every commit triggers a release?)

2) Are there "exceptions" to the list in elpa-packages? That is, if a
package's repo is listed nil in elpa-packages, is it guaranteed that a
branch will exist for that package in ELPA's Git repo?

Thanks,
Mauricio

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Stefan Monnier]

> Thanks for the reply! The particular use case here is to have a
> reproducible environment via the Nix package manager.

I see, yes that makes sense.  Hmm...

> months, it's fair to say a "year's worth" of stability would reduce
> user-facing problems to almost zero -- and I guess archiving 20 releases
> provides that with room to spare.

Actually, 20 is not necessarily that generous in this regard: it's 20
total, but we try to preserve "key" releases (e.g. the ones where the
leading number increased and the ones just before that) rather than
focus only on the most recent ones.  So if the latest version is 6.7,
it's quite possible that 6.4 came out fairly recently but has already
been pruned because we preferred to keep some older ones instead.

If you can work from the elpa.git instead, then you'll avoid those
problems (but the content is slightly different, so it might be less
convenient).

> If it's possible to do this for any package, then this is a great
> alternative! I see there's an elpa-packages file in the ELPA Git repo
> pointing to the sources for each package, which is definitely a format
> that Nix can work with. A few questions, just to be sure:

The URL just points to the "expected" upstream location.  The GNU ELPA
packages are never built from the data at that URL but from the (more or
less) copies we keep in the branches in `elpa.git`.

> 1) When does a commit to the package's repo generate a new release on
> ELPA? (every commit triggers a release?)

When the commit changes the `Version:` header in the main file.

> 2) Are there "exceptions" to the list in elpa-packages? That is, if a
> package's repo is listed nil in elpa-packages, is it guaranteed that a
> branch will exist for that package in ELPA's Git repo?

Regardless of the `:url`, the package's code is kept in the
corresponding `externals/[PKGNAME]` branch.


        Stefan

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Currently, there's no way to get a permanent link to a package version
>>> that happens to be the current one.
>
> FWIW, the situation is even worse because there's no way to get
> a permanent link at all: while we do keep some old versions, we don't
> keep them all.  Currently, the limit is set at 20, meaning that we keep
> up to 20 old versions around per package, and once we hit that limit we
> start pruning the old versions according to a heuristic that intends to
> guess which versions are more important to keep.

Why not say, in addition to the above: we never delete anything more
recent than N years or months?

bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions

[Stefan Monnier]

>>>> Currently, there's no way to get a permanent link to a package version
>>>> that happens to be the current one.
>>
>> FWIW, the situation is even worse because there's no way to get
>> a permanent link at all: while we do keep some old versions, we don't
>> keep them all.  Currently, the limit is set at 20, meaning that we keep
>> up to 20 old versions around per package, and once we hit that limit we
>> start pruning the old versions according to a heuristic that intends to
>> guess which versions are more important to keep.
>
> Why not say, in addition to the above: we never delete anything more
> recent than N years or months?

Yes, that's probably what we should do (tho only for the release
tarballs rather than the devel tarballs).
Patch welcome (the relevant code is in `elpaa--prune-old-tarballs` tho
changes elsewhere will be needed to propagate whether we're pruning
release tarballs or devel tarballs).


        Stefan