unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
@ 2021-02-11 19:46 Mauricio Collares
  2021-02-12 17:32 ` Basil L. Contovounesios
  0 siblings, 1 reply; 10+ messages in thread
From: Mauricio Collares @ 2021-02-11 19:46 UTC (permalink / raw)
  To: 46441

Currently, there's no way to get a permanent link to a package version
that happens to be the current one. For example, auctex is currently at
version 13.0.4; today I can download it from
https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
working as soon as a new version of auctex is released and the old
version gets compressed. This makes it slightly annoying to pin a
particular version of an ELPA package by URL.

I would like to ask the GNU ELPA maintainers to host both
https://elpa.gnu.org/packages/PACKAGE-CURRENTVERSION.lz (permanent) as
well as https://elpa.gnu.org/packagesa/PACKAGE-CURRENTVERSION
(temporary). This would make it easier for people to build reproducible
environments by pinning a package version. I don't particularly care
about exposing it as a link on the website; for my purposes, it's enough
that the file exists. This is perhaps a little bit wasteful, but
compressed versions shouldn't be too big.

Best,
Mauricio





^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-11 19:46 bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions Mauricio Collares
@ 2021-02-12 17:32 ` Basil L. Contovounesios
  2021-02-12 18:10   ` Stefan Monnier
  2021-02-12 18:41   ` Mauricio Collares
  0 siblings, 2 replies; 10+ messages in thread
From: Basil L. Contovounesios @ 2021-02-12 17:32 UTC (permalink / raw)
  To: Mauricio Collares; +Cc: 46441, Stefan Monnier

[CCing the GNU ELPA maintainer.]

Mauricio Collares <mauricio@collares.org> writes:

> Currently, there's no way to get a permanent link to a package version
> that happens to be the current one. For example, auctex is currently at
> version 13.0.4; today I can download it from
> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
> working as soon as a new version of auctex is released and the old
> version gets compressed. This makes it slightly annoying to pin a
> particular version of an ELPA package by URL.
>
> I would like to ask the GNU ELPA maintainers to host both
> https://elpa.gnu.org/packages/PACKAGE-CURRENTVERSION.lz (permanent) as
> well as https://elpa.gnu.org/packagesa/PACKAGE-CURRENTVERSION
> (temporary). This would make it easier for people to build reproducible
> environments by pinning a package version. I don't particularly care
> about exposing it as a link on the website; for my purposes, it's enough
> that the file exists. This is perhaps a little bit wasteful, but
> compressed versions shouldn't be too big.

Wouldn't it be wasteless if the "current version" URL was symbolic and
resolved to the concrete versioned release?

Thanks,

-- 
Basil





^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-12 17:32 ` Basil L. Contovounesios
@ 2021-02-12 18:10   ` Stefan Monnier
  2021-02-12 18:36     ` Mauricio Collares
  2021-04-04  1:00     ` Stefan Kangas
  2021-02-12 18:41   ` Mauricio Collares
  1 sibling, 2 replies; 10+ messages in thread
From: Stefan Monnier @ 2021-02-12 18:10 UTC (permalink / raw)
  To: Basil L. Contovounesios; +Cc: 46441, Mauricio Collares

>> Currently, there's no way to get a permanent link to a package version
>> that happens to be the current one.

FWIW, the situation is even worse because there's no way to get
a permanent link at all: while we do keep some old versions, we don't
keep them all.  Currently, the limit is set at 20, meaning that we keep
up to 20 old versions around per package, and once we hit that limit we
start pruning the old versions according to a heuristic that intends to
guess which versions are more important to keep.

So that makes me question your "upstream" need.

>> For example, auctex is currently at
>> version 13.0.4; today I can download it from
>> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
>> working as soon as a new version of auctex is released and the old
>> version gets compressed. This makes it slightly annoying to pin a
>> particular version of an ELPA package by URL.

Could you expand a bit on why you need to keep references to old
versions, and why you decided to use URLs for that?
That will hopefully help us see what is the best course of action
from here.


        Stefan "not opposed to building the .lz eagerly"






^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-12 18:10   ` Stefan Monnier
@ 2021-02-12 18:36     ` Mauricio Collares
  2021-02-12 19:08       ` Basil L. Contovounesios
  2021-02-12 22:26       ` Stefan Monnier
  2021-04-04  1:00     ` Stefan Kangas
  1 sibling, 2 replies; 10+ messages in thread
From: Mauricio Collares @ 2021-02-12 18:36 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Basil L. Contovounesios, 46441


Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Currently, there's no way to get a permanent link to a package version
>>> that happens to be the current one.
>
> FWIW, the situation is even worse because there's no way to get
> a permanent link at all: while we do keep some old versions, we don't
> keep them all.  Currently, the limit is set at 20, meaning that we keep
> up to 20 old versions around per package, and once we hit that limit we
> start pruning the old versions according to a heuristic that intends to
> guess which versions are more important to keep.
>
> So that makes me question your "upstream" need.
>
>>> For example, auctex is currently at
>>> version 13.0.4; today I can download it from
>>> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
>>> working as soon as a new version of auctex is released and the old
>>> version gets compressed. This makes it slightly annoying to pin a
>>> particular version of an ELPA package by URL.
>
> Could you expand a bit on why you need to keep references to old
> versions, and why you decided to use URLs for that?
> That will hopefully help us see what is the best course of action
> from here.

Hi Stefan,

Thanks for the reply! The particular use case here is to have a
reproducible environment via the Nix package manager. Omitting
irrelevant details, Nix is a source-based package manager whose
"recipes" for building packages typically start by doing the equivalent
of "download this file with this sha256 from this URL" (actually, this
is typically done by fetching a specific commit from a repository, but
in this case I believe tarballs are the only option); that is, all
packages are pinned. This is done for emacs packages too, so it is
possible to have the same exact emacs setup on several machines or to
recover the exact same state if you have to reinstall everything from
scratch for some reason.

No one that uses Nix expects things to work "forevermore" since source
tarballs frequently disappear (people are working on that, though [0]),
but frequently-updated ELPA packages basically "break" reproducibility
every week. More concretely, if I commit my Nix configuration to a Git
repo and a co-worker wants to use it a week from now, it's likely that
the auctex fetch step will fail. Since NixOS releases happen every six
months, it's fair to say a "year's worth" of stability would reduce
user-facing problems to almost zero -- and I guess archiving 20 releases
provides that with room to spare.

Best,
Mauricio

[0] https://www.tweag.io/blog/2020-06-18-software-heritage/





^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-12 17:32 ` Basil L. Contovounesios
  2021-02-12 18:10   ` Stefan Monnier
@ 2021-02-12 18:41   ` Mauricio Collares
  1 sibling, 0 replies; 10+ messages in thread
From: Mauricio Collares @ 2021-02-12 18:41 UTC (permalink / raw)
  To: Basil L. Contovounesios; +Cc: 46441, Stefan Monnier


Basil L. Contovounesios <contovob@tcd.ie> writes:

> [CCing the GNU ELPA maintainer.]
>
> Mauricio Collares <mauricio@collares.org> writes:
>
>> Currently, there's no way to get a permanent link to a package version
>> that happens to be the current one. For example, auctex is currently at
>> version 13.0.4; today I can download it from
>> https://elpa.gnu.org/packages/auctex-13.0.4.tar, but this link will stop
>> working as soon as a new version of auctex is released and the old
>> version gets compressed. This makes it slightly annoying to pin a
>> particular version of an ELPA package by URL.
>>
>> I would like to ask the GNU ELPA maintainers to host both
>> https://elpa.gnu.org/packages/PACKAGE-CURRENTVERSION.lz (permanent) as
>> well as https://elpa.gnu.org/packagesa/PACKAGE-CURRENTVERSION
>> (temporary). This would make it easier for people to build reproducible
>> environments by pinning a package version. I don't particularly care
>> about exposing it as a link on the website; for my purposes, it's enough
>> that the file exists. This is perhaps a little bit wasteful, but
>> compressed versions shouldn't be too big.
>
> Wouldn't it be wasteless if the "current version" URL was symbolic and
> resolved to the concrete versioned release?


Hi Basil,

Thanks for taking the time to reply! I think my use of "CURRENTVERSION"
in the previous email URL was ambiguous. Concretely, what I want to have
is to be able to fetch auctex 13.0.4 (which, as of 2021-02-12 is the
current version) through a URL that will still serve the exact same file
(auctex 13.0.4, and ideally same sha256) a year from now. If I
understand correctly, your suggestion gives a link that always points to
the newest version, due to ambiguity in my previous email. I apologize
for the confusion.

Best,
Mauricio





^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-12 18:36     ` Mauricio Collares
@ 2021-02-12 19:08       ` Basil L. Contovounesios
  2021-02-12 19:31         ` Mauricio Collares
  2021-02-12 22:26       ` Stefan Monnier
  1 sibling, 1 reply; 10+ messages in thread
From: Basil L. Contovounesios @ 2021-02-12 19:08 UTC (permalink / raw)
  To: Mauricio Collares; +Cc: 46441, Stefan Monnier

Mauricio Collares <mauricio@collares.org> writes:

> Thanks for the reply! The particular use case here is to have a
> reproducible environment via the Nix package manager. Omitting
> irrelevant details, Nix is a source-based package manager whose
> "recipes" for building packages typically start by doing the equivalent
> of "download this file with this sha256 from this URL" (actually, this
> is typically done by fetching a specific commit from a repository, but
> in this case I believe tarballs are the only option);

What are the obstacles to fetching from elpa.git here?
E.g. http://git.savannah.gnu.org/cgit/emacs/elpa.git/?h=externals/auctex

Thanks,

-- 
Basil





^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-12 19:08       ` Basil L. Contovounesios
@ 2021-02-12 19:31         ` Mauricio Collares
  0 siblings, 0 replies; 10+ messages in thread
From: Mauricio Collares @ 2021-02-12 19:31 UTC (permalink / raw)
  To: Basil L. Contovounesios; +Cc: 46441, Stefan Monnier

Basil L. Contovounesios <contovob@tcd.ie> writes:

> Mauricio Collares <mauricio@collares.org> writes:
>
>> Thanks for the reply! The particular use case here is to have a
>> reproducible environment via the Nix package manager. Omitting
>> irrelevant details, Nix is a source-based package manager whose
>> "recipes" for building packages typically start by doing the equivalent
>> of "download this file with this sha256 from this URL" (actually, this
>> is typically done by fetching a specific commit from a repository, but
>> in this case I believe tarballs are the only option);
>
> What are the obstacles to fetching from elpa.git here?
> E.g. http://git.savannah.gnu.org/cgit/emacs/elpa.git/?h=externals/auctex

Hi Basil,

If it's possible to do this for any package, then this is a great
alternative! I see there's an elpa-packages file in the ELPA Git repo
pointing to the sources for each package, which is definitely a format
that Nix can work with. A few questions, just to be sure:

1) When does a commit to the package's repo generate a new release on
ELPA? (every commit triggers a release?)

2) Are there "exceptions" to the list in elpa-packages? That is, if a
package's repo is listed nil in elpa-packages, is it guaranteed that a
branch will exist for that package in ELPA's Git repo?

Thanks,
Mauricio






^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-12 18:36     ` Mauricio Collares
  2021-02-12 19:08       ` Basil L. Contovounesios
@ 2021-02-12 22:26       ` Stefan Monnier
  1 sibling, 0 replies; 10+ messages in thread
From: Stefan Monnier @ 2021-02-12 22:26 UTC (permalink / raw)
  To: Mauricio Collares; +Cc: Basil L. Contovounesios, 46441

> Thanks for the reply! The particular use case here is to have a
> reproducible environment via the Nix package manager.

I see, yes that makes sense.  Hmm...

> months, it's fair to say a "year's worth" of stability would reduce
> user-facing problems to almost zero -- and I guess archiving 20 releases
> provides that with room to spare.

Actually, 20 is not necessarily that generous in this regard: it's 20
total, but we try to preserve "key" releases (e.g. the ones where the
leading number increased and the ones just before that) rather than
focus only on the most recent ones.  So if the latest version is 6.7,
it's quite possible that 6.4 came out fairly recently but has already
been pruned because we preferred to keep some older ones instead.

If you can work from the elpa.git instead, then you'll avoid those
problems (but the content is slightly different, so it might be less
convenient).

> If it's possible to do this for any package, then this is a great
> alternative! I see there's an elpa-packages file in the ELPA Git repo
> pointing to the sources for each package, which is definitely a format
> that Nix can work with. A few questions, just to be sure:

The URL just points to the "expected" upstream location.  The GNU ELPA
packages are never built from the data at that URL but from the (more or
less) copies we keep in the branches in `elpa.git`.

> 1) When does a commit to the package's repo generate a new release on
> ELPA? (every commit triggers a release?)

When the commit changes the `Version:` header in the main file.

> 2) Are there "exceptions" to the list in elpa-packages? That is, if a
> package's repo is listed nil in elpa-packages, is it guaranteed that a
> branch will exist for that package in ELPA's Git repo?

Regardless of the `:url`, the package's code is kept in the
corresponding `externals/[PKGNAME]` branch.


        Stefan






^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-02-12 18:10   ` Stefan Monnier
  2021-02-12 18:36     ` Mauricio Collares
@ 2021-04-04  1:00     ` Stefan Kangas
  2021-04-04 23:09       ` Stefan Monnier
  1 sibling, 1 reply; 10+ messages in thread
From: Stefan Kangas @ 2021-04-04  1:00 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Basil L. Contovounesios, 46441, Mauricio Collares

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Currently, there's no way to get a permanent link to a package version
>>> that happens to be the current one.
>
> FWIW, the situation is even worse because there's no way to get
> a permanent link at all: while we do keep some old versions, we don't
> keep them all.  Currently, the limit is set at 20, meaning that we keep
> up to 20 old versions around per package, and once we hit that limit we
> start pruning the old versions according to a heuristic that intends to
> guess which versions are more important to keep.

Why not say, in addition to the above: we never delete anything more
recent than N years or months?





^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions
  2021-04-04  1:00     ` Stefan Kangas
@ 2021-04-04 23:09       ` Stefan Monnier
  0 siblings, 0 replies; 10+ messages in thread
From: Stefan Monnier @ 2021-04-04 23:09 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: Basil L. Contovounesios, 46441, Mauricio Collares

>>>> Currently, there's no way to get a permanent link to a package version
>>>> that happens to be the current one.
>>
>> FWIW, the situation is even worse because there's no way to get
>> a permanent link at all: while we do keep some old versions, we don't
>> keep them all.  Currently, the limit is set at 20, meaning that we keep
>> up to 20 old versions around per package, and once we hit that limit we
>> start pruning the old versions according to a heuristic that intends to
>> guess which versions are more important to keep.
>
> Why not say, in addition to the above: we never delete anything more
> recent than N years or months?

Yes, that's probably what we should do (tho only for the release
tarballs rather than the devel tarballs).
Patch welcome (the relevant code is in `elpaa--prune-old-tarballs` tho
changes elsewhere will be needed to propagate whether we're pruning
release tarballs or devel tarballs).


        Stefan






^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-04-04 23:09 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-11 19:46 bug#46441: GNU ELPA feature request: host .lz archives (as well as uncompressed) for current versions Mauricio Collares
2021-02-12 17:32 ` Basil L. Contovounesios
2021-02-12 18:10   ` Stefan Monnier
2021-02-12 18:36     ` Mauricio Collares
2021-02-12 19:08       ` Basil L. Contovounesios
2021-02-12 19:31         ` Mauricio Collares
2021-02-12 22:26       ` Stefan Monnier
2021-04-04  1:00     ` Stefan Kangas
2021-04-04 23:09       ` Stefan Monnier
2021-02-12 18:41   ` Mauricio Collares

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).