[bug#33535] [PATCH] refresh: Account for overlapping updater coverage.

[bug#33535] [PATCH] refresh: Account for overlapping updater coverage.

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

* guix/scripts/refresh.scm (list-updaters-and-exit): Do not assume updater
predicates are disjoint.  Track covered packages directly.
---
Hello Guix,

Some of our packages are covered by more than one of our updaters:

scheme@(guile-user)> ,use(gnu packages)(guix packages)(guix upstream)
scheme@(guile-user)> ,use(srfi srfi-1)(srfi srfi-26)
scheme@(guile-user)> (define updaters (force (@@ (guix upstream) %updaters)))
scheme@(guile-user)> (define predicates (map upstream-updater-predicate updaters))
scheme@(guile-user)> (define doubles
  (fold-packages
    (lambda (pkg result)
      (if (> (count (cut <> pkg) predicates) 1)
         (cons pkg result)
       result))
    '()))
scheme@(guile-user)> (length doubles)
$1 = 469
scheme@(guile-user)> (map package-name doubles)
$2 = ("agda" "emacs-agda2-mode" "fribidi" "raincat" "ghc-tasty-quickcheck" ... "ghc-hxt")

It seams mostly packages covered by both the "hackage" and "stackage"
updaters.  And Fribidi is a GNU package but hosted on github.

Currently this leads to double-counting while computing total package
coverage and a too optimistic result (by about 5.4%).

The below patch fixes it by tracking the (un)covered packages directly.


 guix/scripts/refresh.scm | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index 58fc64db1..f7d2cffb7 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -179,24 +179,24 @@ specified with `--select'.\n"))
 
   (let* ((packages (fold-packages cons '()))
          (total    (length packages)))
-    (define covered
-      (fold (lambda (updater covered)
-              (let ((matches (count (upstream-updater-predicate updater)
-                                    packages)))
+    (define uncovered
+      (fold (lambda (updater uncovered)
+              (let ((matches (filter (upstream-updater-predicate updater)
+                                     packages)))
                 ;; TRANSLATORS: The parenthetical expression here is rendered
                 ;; like "(42% coverage)" and denotes the fraction of packages
                 ;; covered by the given updater.
                 (format #t (G_ "  - ~a: ~a (~2,1f% coverage)~%")
                         (upstream-updater-name updater)
                         (G_ (upstream-updater-description updater))
-                        (* 100. (/ matches total)))
-                (+ covered matches)))
-            0
+                        (* 100. (/ (length matches) total)))
+                (lset-difference eq? uncovered matches)))
+            packages
             (force %updaters)))
 
     (newline)
     (format #t (G_ "~2,1f% of the packages are covered by these updaters.~%")
-            (* 100. (/ covered total))))
+            (* 100. (/ (- total (length uncovered)) total))))
   (exit 0))
 
 (define (warn-no-updater package)
-- 
2.19.1

[bug#33535] [PATCH] refresh: Account for overlapping updater coverage.

[Ludovic Courtès]

Hello!

ericbavier@centurylink.net skribis:

> From: Eric Bavier <bavier@member.fsf.org>
>
> * guix/scripts/refresh.scm (list-updaters-and-exit): Do not assume updater
> predicates are disjoint.  Track covered packages directly.
> ---
> Hello Guix,
>
> Some of our packages are covered by more than one of our updaters:
>
> scheme@(guile-user)> ,use(gnu packages)(guix packages)(guix upstream)
> scheme@(guile-user)> ,use(srfi srfi-1)(srfi srfi-26)
> scheme@(guile-user)> (define updaters (force (@@ (guix upstream) %updaters)))
> scheme@(guile-user)> (define predicates (map upstream-updater-predicate updaters))
> scheme@(guile-user)> (define doubles
>   (fold-packages
>     (lambda (pkg result)
>       (if (> (count (cut <> pkg) predicates) 1)
>          (cons pkg result)
>        result))
>     '()))
> scheme@(guile-user)> (length doubles)
> $1 = 469
> scheme@(guile-user)> (map package-name doubles)
> $2 = ("agda" "emacs-agda2-mode" "fribidi" "raincat" "ghc-tasty-quickcheck" ... "ghc-hxt")
>
> It seams mostly packages covered by both the "hackage" and "stackage"
> updaters.  And Fribidi is a GNU package but hosted on github.
>
> Currently this leads to double-counting while computing total package
> coverage and a too optimistic result (by about 5.4%).
>
> The below patch fixes it by tracking the (un)covered packages directly.

Oh, good catch.  LGTM!

That also means we’ll have to work on our updaters…  :-)

Thanks,
Ludo’.

bug#33535: [PATCH] refresh: Account for overlapping updater coverage.

[Eric Bavier]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

On Thu, 29 Nov 2018 18:42:38 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Hello!
> 
> ericbavier@centurylink.net skribis:
> 
> > From: Eric Bavier <bavier@member.fsf.org>
> >
> > It seams mostly packages covered by both the "hackage" and "stackage"
> > updaters.  And Fribidi is a GNU package but hosted on github.
> >
> > Currently this leads to double-counting while computing total package
> > coverage and a too optimistic result (by about 5.4%).
> >
> > The below patch fixes it by tracking the (un)covered packages directly.  
> 
> Oh, good catch.  LGTM!

Applied in cba7ddcf603455c6692eb50c8bbf203a6bf17ab1

> 
> That also means we’ll have to work on our updaters…  :-)
> 

Yup.  

I have a prototype for an "arch" importer that parses the Archlinux
PKGBUILD files (32% coverage even without any fancy package name
mapping).

I also have a patch to the github updater so that it can update
packages that use git-fetch.  This brings the coverage up to 15.9% from
9.6%.  I think this will be useful if we keep moving packages away from
github's generated tarballs.

`~Eric

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#33535] [PATCH] refresh: Account for overlapping updater coverage.

[swedebugia]

On 2018-12-11 05:59, Eric Bavier wrote:
snip

> I have a prototype for an "arch" importer that parses the Archlinux
> PKGBUILD files (32% coverage even without any fancy package name
> mapping).

That sounds nice Eric. Can I see code somewhere? :)

-- 
Cheers 
Swedebugia

[bug#33535] [PATCH] refresh: Account for overlapping updater coverage.

[Eric Bavier]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

On Wed, 12 Dec 2018 03:11:44 -0800
swedebugia@riseup.net wrote:

> On 2018-12-11 05:59, Eric Bavier wrote:
> snip
> 
> > I have a prototype for an "arch" importer that parses the Archlinux
> > PKGBUILD files (32% coverage even without any fancy package name
> > mapping).  
> 
> That sounds nice Eric. Can I see code somewhere? :)
> 

It's on my "feature/arch-updater" branch:

  https://notabug.org/bavier/guix/src/feature/arch-updater

The current commit:

  https://notabug.org/bavier/guix/commit/7d711d622db43bc2a9074d494f8cb11e6a9291c8

I'm open to suggestions :)

`~Eric

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]