Repology and outdated packages

Repology and outdated packages

[kiasoc5]

Dear Guix,

I've been watching the Repology page for Guix and I've noticed that we've dropped to 51% outdated packages [https://repology.org/repository/gnuguix]. We used to be at 40% outdated packages a few months ago.

I know that rolling release distros don't have to have the latest packages but this is not the best representation of Guix, especially when our friend NixOS claims they have the largest and most updated package collection [https://nixos.org/blog/announcements.html#nixos-22.05].

So without demanding more maintainer time, for now I just convince myself that:
- key toolchains such as Rust and Go are not always up to date, thus blocking the upgrade of several packages
- we package their dependencies separately (eg Rust crates, Go modules), these are a significant portion of Guix that cannot be constantly updated easily
- the Guix 1.4 release is coming soon so more time may be spent debugging that instead of updating the package archives
- the Guix branching model of staging/core-updates means we could say Guix is an LTS rolling release distro, which there aren't that many of

However I hope that faster package reviews can mean we stay updated faster without compromising package quality.

WDYT?

Re: Repology and outdated packages

[Nicolas Goaziou]

Hello,

kiasoc5@disroot.org writes:

> I've been watching the Repology page for Guix and I've noticed that
> we've dropped to 51% outdated packages
> [https://repology.org/repository/gnuguix]. We used to be at 40%
> outdated packages a few months ago.

Repology hasn't been able to caught Guix package updates for a while
now. As a consequence, many packages are marked as outdated in Repology
even though they are not.

Regards,
-- 
Nicolas Goaziou

Re: Repology and outdated packages

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

kiasoc5@disroot.org schreef op di 07-06-2022 om 18:39 [+0000]:
> So without demanding more maintainer time, for now I just convince myself that:
> - key toolchains such as Rust and Go are not always up to date, thus blocking the
>  upgrade of several packages

In my experience with antioxidant, not-up-to-date Rust toolchains do
not prevent upgrading packages -- in the process of resolving build
failures, I updated some crates.  This never resulted in a ‘you need an
new toolchain error’.  At worst, I had to disable the "unstable",
"nightly" or "generic-simd" features of the crate.

There are a lot of outdated crates in Guix, but AFAICT this is
unrelated to the newness of the toolchain.

I don't know about Go.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Repology and outdated packages

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

kiasoc5@disroot.org schreef op di 07-06-2022 om 18:39 [+0000]:
> - we package their dependencies separately (eg Rust crates, Go modules), these are a significant portion of Guix that cannot be constantly updated easily
> - the Guix 1.4 release is coming soon so more time may be spent debugging that instead of updating the package archives
[...]
> However I hope that faster package reviews can mean we stay updated faster without compromising package quality.

FWIW, antioxidant removes most of the complexity of Rust's dependency
system, which should help with packaging and reviewing.  No need to
package old versions to satisfy Cargo and making sure the old version
builds, has an appropriate synopsis/description, doesn't introduce
malware, ... Theoretically (currently untested!) faster build times, no
need to package Windows or redox crates.

The cost is that, as you seem to write, that the dependent might not
support the old version of the dependency, but updating is something
upstream needs to do eventually anyway, and (antioxidant) in theory
(untested) supports working with multiple versions.  There's also the
option of making some compatibility changes in the Guix version of the
dependency (undoing a removal of a function, or adding some aliases,
etc.).

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Repology and outdated packages

[Ludovic Courtès]

Hi!

kiasoc5@disroot.org skribis:

> I know that rolling release distros don't have to have the latest packages but this is not the best representation of Guix, especially when our friend NixOS claims they have the largest and most updated package collection [https://nixos.org/blog/announcements.html#nixos-22.05].

Guix is *potentially* even more up-to-date than NixOS thanks to
‘--with-latest’ and ‘--with-branch’!  \o/

  https://guix.gnu.org/manual/devel/en/html_node/Package-Transformation-Options.html

Seriously though, we could take better advantage of the tooling that we
have: ‘guix refresh’, ‘guix graph’, and the corresponding APIs.  With
that, we can write code that automatically tries out package updates and
prepares patches, for instance.  We could even largely automate “update
trains” (what we’re doing with master/staging/core-updates).

Ideas for a motivated hacker!

Ludo’.

Re: Repology and outdated packages

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

[…]

> Seriously though, we could take better advantage of the tooling that we
> have: ‘guix refresh’, ‘guix graph’, and the corresponding APIs.  With
> that, we can write code that automatically tries out package updates and
> prepares patches, for instance.  We could even largely automate “update
> trains” (what we’re doing with master/staging/core-updates).

Maybe it's ok to push directly packages with #:tests #true and a low
number of triggered builds by the update.

Also, we probably should run the make check-system TESTS="XYZ" tests if
a service depends on an updated package.

Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Repology and outdated packages

[kiasoc5]

June 7, 2022 at 6:47 PM, "Nicolas Goaziou" <mail@nicolasgoaziou.fr mailto:mail@nicolasgoaziou.fr?to=%22Nicolas%20Goaziou%22%20%3Cmail%40nicolasgoaziou.fr%3E > wrote:

> Repology hasn't been able to caught Guix package updates for a while
> now. As a consequence, many packages are marked as outdated in Repology
> even though they are not.

Another point of Repology is that binary up-to-dateness (latest/not) is less granular than "package is X versions behind latest". Guix is usually only a version or 2 behind in many projects which is fresh enough for me.

Re: Repology and outdated packages

[kiasoc5]

June 8, 2022 at 9:38 PM, "Ludovic Courtès" <ludo@gnu.org mailto:ludo@gnu.org?to=%22Ludovic%20Court%C3%A8s%22%20%3Cludo%40gnu.org%3E > wrote:

> Guix is *potentially* even more up-to-date than NixOS thanks to
> ‘--with-latest’ and ‘--with-branch’! \o/

I do use --with-latest for testing package upgrades. But it is tedious to type

$ guix build X --with-latest=X

in order to test the latest package.

What would be nice is if

$ guix build --latest X

is an equivalent command. Certainly a shell script could take care of that:

guix-build-with-latest() {
    guix build "$1" --with-latest="$1"
}

but having it done in guix itself would be convenient.

> Seriously though, we could take better advantage of the tooling that we
> have: ‘guix refresh’, ‘guix graph’, and the corresponding APIs. With
> that, we can write code that automatically tries out package updates and
> prepares patches, for instance. We could even largely automate “update
> trains” (what we’re doing with master/staging/core-updates).

It could be an option to guix refresh, for example

$ guix build --latest X --export-patch

would build X with the latest version and export a patch with an appropriate commit message (like marking failed builds with DRAFT). This would help users contribute without having to clone the git repo.

Re: Repology and outdated packages

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

Maxime Devos schreef op di 07-06-2022 om 23:32 [+0200]:
> In my experience with antioxidant, not-up-to-date Rust toolchains do
> not prevent upgrading packages -- in the process of resolving build
> failures, I updated some crates.  This never resulted in a ‘you need
> an
> new toolchain error’.  At worst, I had to disable the "unstable",
> "nightly" or "generic-simd" features of the crate.

Correction (adding the rust-git-path crate to update another crate to
avoid non-building crate due to updated crates with a different API):

Saving crate informtion in /gnu/store/fr292pqsfas2i9nwc3bijx5skb8g3ayg-rust-git-path-0.1.3/lib/guixcrate/git-path.crate-info
error[E0658]: use of unstable library feature 'is_symlink'
  --> src/realpath.rs:50:34
   |
50 |                     if real_path.is_symlink() {
   |                                  ^^^^^^^^^^
   |
   = note: see issue #85748 <https://github.com/rust-lang/rust/issues/85748> for more information


Looks like it is possible to work-around avoid though:

https://github.com/rust-lang/cargo/commit/c6745a3d7fcea3a949c3e13e682b8ddcbd213add

Alternatively, I could set RUSTC_BOOTSTRAP=1.  From
<https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html>

The build system sets RUSTC_BOOTSTRAP=1. This special variable means
to break the stability guarantees of rust: Allow using #![feature(...)]
with a compiler that's not nightly. This should never be used except
when bootstrapping the compiler.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Repology and outdated packages

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

On 2022-06-07, Nicolas Goaziou wrote:
> kiasoc5@disroot.org writes:
>
>> I've been watching the Repology page for Guix and I've noticed that
>> we've dropped to 51% outdated packages
>> [https://repology.org/repository/gnuguix]. We used to be at 40%
>> outdated packages a few months ago.
>
> Repology hasn't been able to caught Guix package updates for a while
> now. As a consequence, many packages are marked as outdated in Repology
> even though they are not.

I wondered about this again, and was pointed to:

  https://github.com/repology/repology-updater/issues/1261

Appears that where the website is hosted might indirectly (e.g. nobody
explicitly configured this on guix infrastructure) have traffic
originating in Russian blocked...

Wonder if it makes sense to sync the files to multiple sites and put in
round-robin DNS? Apparently it is already available at bayfront.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Repology and outdated packages

[Maxim Cournoyer]

Hi,

Vagrant Cascadian <vagrant@debian.org> writes:

> On 2022-06-07, Nicolas Goaziou wrote:
>> kiasoc5@disroot.org writes:
>>
>>> I've been watching the Repology page for Guix and I've noticed that
>>> we've dropped to 51% outdated packages
>>> [https://repology.org/repository/gnuguix]. We used to be at 40%
>>> outdated packages a few months ago.
>>
>> Repology hasn't been able to caught Guix package updates for a while
>> now. As a consequence, many packages are marked as outdated in Repology
>> even though they are not.
>
> I wondered about this again, and was pointed to:
>
>   https://github.com/repology/repology-updater/issues/1261
>
> Appears that where the website is hosted might indirectly (e.g. nobody
> explicitly configured this on guix infrastructure) have traffic
> originating in Russian blocked...
>
> Wonder if it makes sense to sync the files to multiple sites and put in
> round-robin DNS? Apparently it is already available at bayfront.

Assuming the website fall-back that exists on bayfront is complete,
that'd be worth it I believe.  If you're comfortable with DNS
configuration, send a patch effecting the change to the guix-maintenance
repo to guix-sysadmins@gnu.org for review!

Thanks,

Maxim