Shall updaters fall back to other updaters?

Shall updaters fall back to other updaters?

[Hartmut Goebel]

Hi,

while working on refreshing to a specific version (see 
https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00222.html) I 
discovered that the updaters fall back to another updater. Is this intended?

Concrete example (using refresh to a specific version): Package "xlsxio" 
has no version 0.2.30. When trying to refresh to this version, the 
github updater comes first and of course fails to get this version. Then 
the generic-git updater is triggered and tries to get the version.

IMHO each package should be handled by a single updater.

What do you think?

BTW 1: There are other packages which are handled be several updaters: 
If you sum up the percent valued of "guix refresh --list-updaters" you 
will get about 140%. Anyhow the generic-git updater contributed with 
about 30% to this amount.

BTW 2: Which updater is used for each package is non-deterministic.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Shall updaters fall back to other updaters?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1298 bytes --]

Hartmut Goebel schreef op do 30-06-2022 om 10:58 [+0200]:
> Hi,
> 
> while working on refreshing to a specific version (see 
> https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00222.html) I 
> discovered that the updaters fall back to another updater. Is this intended?
> 
> Concrete example (using refresh to a specific version): Package "xlsxio" 
> has no version 0.2.30. When trying to refresh to this version, the 
> github updater comes first and of course fails to get this version. Then 
> the generic-git updater is triggered and tries to get the version.
> 
> IMHO each package should be handled by a single updater.

I think that, in the absence of pre-existing knowledge which updater is
the right one, trying out several until we get the ‘right’ one is
reasonable.

However, to avoid non-determinism, some CPU time, some I/O and
messiness, I think that somehow annotating packages to write down
_which_ updater applies would be reasonable (maybe with some defaults,
e.g. for minetest mods, ContentDB would be considered authoritive)

Anyway, this idea of ‘authoritive updaters’ as it has been raised
before (I think by Liliana, in the context of the Minetest updater and
generic-git), but I couldn't find the mail again.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Shall updaters fall back to other updaters?

[Ludovic Courtès]

Hello,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> while working on refreshing to a specific version (see
> https://lists.gnu.org/archive/html/guix-devel/2022-06/msg00222.html) I
> discovered that the updaters fall back to another updater. Is this
> intended?

Yes.

> Concrete example (using refresh to a specific version): Package
> "xlsxio" has no version 0.2.30. When trying to refresh to this
> version, the github updater comes first and of course fails to get
> this version. Then the generic-git updater is triggered and tries to
> get the version.
>
> IMHO each package should be handled by a single updater.

I agree with Maxime: trying out several updaters is the right one.

> What do you think?
>
> BTW 1: There are other packages which are handled be several updaters:
> If you sum up the percent valued of "guix refresh --list-updaters" you
> will get about 140%. Anyhow the generic-git updater contributed with
> about 30% to this amount.

Heh, true.  :-)

> BTW 2: Which updater is used for each package is non-deterministic.

Do you have an example?  I’d think they’re always tried in the same
order, no?

Thanks,
Ludo’.

Re: Shall updaters fall back to other updaters?

[Hartmut Goebel]

Am 01.07.22 um 15:15 schrieb Ludovic Courtès:
>> BTW 2: Which updater is used for each package is non-deterministic.
> Do you have an example?  I’d think they’re always tried in the same
> order, no?

When looking at the code, I don't see any means of sorting: 
<https://git.savannah.gnu.org/cgit/guix.git/tree/guix/upstream.scm#n245> 
and below. (Maybe sorting is hidden in one of the function 
guile-internal used there there.)


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Shall updaters fall back to other updaters?

[Kaelyn]

------- Original Message -------
On Sunday, July 3rd, 2022 at 1:12 AM, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:


> Am 01.07.22 um 15:15 schrieb Ludovic Courtès:
>
> > > BTW 2: Which updater is used for each package is non-deterministic.
> > > Do you have an example? I’d think they’re always tried in the same
> > > order, no?
>
>
> When looking at the code, I don't see any means of sorting:
> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/upstream.scm#n245
>
> and below. (Maybe sorting is hidden in one of the function
> guile-internal used there there.)

I just looked at 'importer-modules and did some tracing of where the list of modules come from (back through 'all-modules, etc). It appears that by default the list of importer modules comes from 'scheme-file, which is defined at:

https://git.savannah.gnu.org/cgit/guix.git/tree/guix/discovery.scm#n43

That function builds a list of scheme files found recursively under a given directory with the help of 'scandir*. There is no sorting there or along the call chain back to 'importer-modules, so I'm fairly sure the directory entries are returned in a filesystem-dependent order. If the filesystem returns entries in sorted order or if the contents of that directory tree are relatively static (in that the order of directory entries hasn't changed), then the results will appear to be in a consistent/deterministic order. To me, this feels like the importers will need a more deterministic order imposed on them to get import results that are consistent across systems. HTH!

Cheers,
Kaelyn

Re: Shall updaters fall back to other updaters?

[Hartmut Goebel]

Am 03.07.22 um 17:11 schrieb Kaelyn:
> To me, this feels like the importers will need a more deterministic order imposed on them to get import results that are consistent across systems. HTH!

Many thanks for your analysis.

Meanwhile it came to my mind that I actually have different 
installations of guix which I can compare, Below are three 
installations. each one has a different order (these are actually 
different versions of guix with different importers/updaters, anyhow the 
order should be the same):

stackage pypi opam minetest launchpad kde hexpm hackage gnome github
generic-git gem elpa egg crate bioconductor cran cpan savannah
generic-html gnu-ftp sourceforge xorg kernel.org gnu

stackage pypi opam minetest launchpad kde hackage gnome github
generic-git gem elpa egg crate cran bioconductor cpan kernel.org
sourceforge gnu generic-html gnu-ftp xorg savannah

stackage pypi opam minetest launchpad kde hackage gnome github
generic-git gem elpa egg crate bioconductor cran cpan sourceforge
generic-html gnu-ftp savannah xorg kernel.org gnu

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Shall updaters fall back to other updaters?

[zimoun]

Hi,

On Thu, 30 Jun 2022 at 10:58, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:

> BTW 2: Which updater is used for each package is non-deterministic.

From (guix discovery), ’all-modules’ is not deterministic because it
depends on how the filesystem tree is walked.

Therefore, indeed ’importer-modules’ from (guix upstream) is not sorted
and the order is non-deterministic.

(Note that ’fold-packages’ from (guix packages) is not determining too,
if I remember correctly.)


Well, make ’all-modules’ deterministic requires some benchmark because
it could like to some time penalty; even if the order does not matter.
Some commands are already enough slow for adding an extra penalty. ;-)

Maybe sorting ’%updaters’ would be enough; something like,

--8<---------------cut here---------------start------------->8---
(define %updaters
  ;; The list of publically-known updaters.
  (delay (sort (fold-module-public-variables (lambda (obj result)
                                               (if (upstream-updater? obj)
                                                   (cons obj result)
                                                   result))
                                             '()
                                             (importer-modules))
               (lambda (u1 u2)
                 (string<?
                  (upstream-updater-name u1) (upstream-updater-name u2))))))
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

Re: Shall updaters fall back to other updaters?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Thu, 30 Jun 2022 at 10:58, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:
>
>> BTW 2: Which updater is used for each package is non-deterministic.
>
> From (guix discovery), ’all-modules’ is not deterministic because it
> depends on how the filesystem tree is walked.

Nope!  ‘all-modules’ *is* deterministic because it builds upon
‘scheme-files’, which is deterministic and documented as such.

> Maybe sorting ’%updaters’ would be enough; something like,
>
> (define %updaters
>   ;; The list of publically-known updaters.
>   (delay (sort (fold-module-public-variables (lambda (obj result)

Whether ‘fold-module-public-variables’ is deterministic depends on
whether ‘module-map’ is deterministic, which in turn depends on
‘hash-map->list’, which is not deterministic AFAICS.

So what we could do is replace uses of ‘module-map’ with a variant that
sorts variables.

That said, the only case where it can influence updater order is when
several of them are defined in the same module.

Thoughts?

Ludo’.

Re: Shall updaters fall back to other updaters?

[zimoun]

Hi,

On mer., 06 juil. 2022 at 16:16, Ludovic Courtès <ludo@gnu.org> wrote:

> Nope!  ‘all-modules’ *is* deterministic because it builds upon
> ‘scheme-files’, which is deterministic and documented as such.

Oh indeed!  My bad, I have overlooked that ’scheme-files’ uses
’scandir*’ instead of Guile scandir; and this scandir* defined in (guix
syscall) sorts the result.


>> Maybe sorting ’%updaters’ would be enough; something like,
>>
>> (define %updaters
>>   ;; The list of publically-known updaters.
>>   (delay (sort (fold-module-public-variables (lambda (obj result)
>
> Whether ‘fold-module-public-variables’ is deterministic depends on
> whether ‘module-map’ is deterministic, which in turn depends on
> ‘hash-map->list’, which is not deterministic AFAICS.
>
> So what we could do is replace uses of ‘module-map’ with a variant that
> sorts variables.
>
> That said, the only case where it can influence updater order is when
> several of them are defined in the same module.

Indeed, and that’s the case, no?  From [1]:

        bioconductor cran 
        savannah generic-html gnu-ftp sourceforge xorg kernel.org gnu

        cran bioconductor
        kernel.org sourceforge gnu generic-html gnu-ftp xorg savannah
       
        bioconductor cran
        sourceforge generic-html gnu-ftp savannah xorg kernel.org gnu

we see that the updaters are file-sorted (removed here) and only they
appear unsorted when defined in the same module (underlined here).


Well, I count 5 calls to ’module-map’.  Do you mean replace all of them
by ’module-map*’ which guarantees a stable order?  Or only ’module-map’
defined in ’fold-module-public-variables’ and
’fold-module-public-variables*’?


1: <https://yhetil.org/guix/9a68beb7-50f0-9998-9daf-2b036a8e9e66@crazy-compilers.com>


Cheers,
simon

Re: Shall updaters fall back to other updaters?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

>>> Maybe sorting ’%updaters’ would be enough; something like,
>>>
>>> (define %updaters
>>>   ;; The list of publically-known updaters.
>>>   (delay (sort (fold-module-public-variables (lambda (obj result)
>>
>> Whether ‘fold-module-public-variables’ is deterministic depends on
>> whether ‘module-map’ is deterministic, which in turn depends on
>> ‘hash-map->list’, which is not deterministic AFAICS.
>>
>> So what we could do is replace uses of ‘module-map’ with a variant that
>> sorts variables.
>>
>> That said, the only case where it can influence updater order is when
>> several of them are defined in the same module.

[...]

> we see that the updaters are file-sorted (removed here) and only they
> appear unsorted when defined in the same module (underlined here).

Yeah, you wort right from the start: we should sort ‘%updaters’.  I’ll
push something along these lines.

(The other option I contemplated was to sort things in
‘fold-module-public-variables’, but that’s potentially costly for things
like packages, with many variables per module, so I thought we’re rather
not do it by default.)

Thanks,
Ludo’.