guix melpa mirror!

guix melpa mirror!

[jgart]

Have people seen this yet?

https://github.com/babariviere/guix-emacs

pretty epic I think

https://github.com/babariviere/guix-emacs/blob/master/docs/how_it_works.org


https://github.com/babariviere/guix-emacs/blob/master/scripts/update

https://github.com/babariviere/guix-emacs/blob/master/emacs/packages/melpa.scm

Re: guix melpa mirror!

[Joshua Branson]

jgart <jgart@dismail.de> writes:

> Have people seen this yet?
>
> https://github.com/babariviere/guix-emacs
>
> pretty epic I think
>
> https://github.com/babariviere/guix-emacs/blob/master/docs/how_it_works.org
>
>
> https://github.com/babariviere/guix-emacs/blob/master/scripts/update
>
> https://github.com/babariviere/guix-emacs/blob/master/emacs/packages/melpa.scm
>

That is pretty rad!  I wonder if guix has packaged all of the elpa
packages?  If not, then this source code might be useful.  :)

Re: guix melpa mirror!

[jgart]

On Wed, 16 Nov 2022 10:52:24 -0500 Joshua Branson <jbranso@dismail.de> wrote:
> That is pretty rad!  I wonder if guix has packaged all of the elpa
> packages?  If not, then this source code might be useful.  :)

There's a similar completist automated project for R/CRAN repos that's
on GitHub by rekado and lars:

https://github.com/guix-science/guix-cran

Re: guix melpa mirror!

[Ludovic Courtès]

jgart <jgart@dismail.de> skribis:

> Have people seen this yet?
>
> https://github.com/babariviere/guix-emacs
>
> pretty epic I think
>
> https://github.com/babariviere/guix-emacs/blob/master/docs/how_it_works.org

Fun!

Would be nice to share more code with (guix import elpa).

Ludo’.

Re: guix melpa mirror!

[jgart]

On Thu, 17 Nov 2022 16:14:41 +0100 Ludovic Courtès <ludo@gnu.org> wrote:
> Fun!
> 
> Would be nice to share more code with (guix import elpa).

Ah, yes that's a great idea!! I hadn't made the association yet between those two.

I'll start reading through them with that in mind.

I would love for guix to have very powerful automated package creation for all language ecosystems!

Long term big picture goal thinking here ;()

elpa's a start

Re: guix melpa mirror!

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 3346 bytes --]

Hi,

On Thu, 17 Nov 2022 at 19:06, jgart <jgart@dismail.de> wrote:

> I would love for guix to have very powerful automated package creation
> for all language ecosystems! 

Well, it strongly depends on the quality of the targeted language
ecosystem.  For some, they provide enough metadata to rely on for good
automatizing; for instance, R with CRAN or Bioconductor.

Sadly, for many others ecosystem, they (upstream) do not provide enough
metadata to automatically fill all the package fields.  And some manual
tweaks are required.

For example, let count the number of packages that are tweaking their
’arguments’ fields (from ’#:tests? #f’ to complex phases modifications).
This is far from being a perfect metrics but it is a rough indication
about upstream quality: if they provide clean package respecting their
build system or if the package requires Guix adjustments.

Well, I get:

      r            : 2093 = 2093 = 1991 + 102 

which is good (only ~5% require ’arguments’ tweaks), but

      python       : 2630 = 2630 = 803  + 1827

is bad (only ~31% do not require an ’arguments’ tweak).

About Emacs, it reads,

      emacs        : 1222 = 1222 = 874  + 348 

From my point of view, it seems that it would be hard to have full
automated Emacs packaging for Guix.  Well, it requires some work to find
some heuristics at importing (converting) time and that’s not
straightforward, IMHO.

Cheers,
simon

--8<---------------cut here---------------start------------->8---
$ guix repl -- arguments-vs-import.scm
key          : tot  = tot  = no-arguments + arguments
ocaml        : 57   = 57   = 0    + 57  
haskell      : 723  = 723  = 505  + 218 
clojure      : 11   = 11   = 0    + 11  
qt           : 226  = 226  = 98   + 128 
copy         : 105  = 105  = 1    + 104 
maven        : 1    = 1    = 0    + 1   
node         : 48   = 48   = 2    + 46  
minetest-mod : 18   = 18   = 18   + 0   
chicken      : 9    = 9    = 0    + 9   
emacs        : 1222 = 1222 = 874  + 348 
linux-module : 14   = 14   = 0    + 14  
raw          : 1    = 1    = 1    + 0   
glib-or-gtk  : 128  = 128  = 31   + 97  
asdf/source  : 659  = 659  = 553  + 106 
dune         : 226  = 226  = 39   + 187 
go           : 483  = 483  = 0    + 483 
cmake        : 1115 = 1115 = 92   + 1023
minify       : 11   = 11   = 1    + 10  
perl         : 823  = 823  = 710  + 113 
android-ndk  : 11   = 11   = 0    + 11  
waf          : 25   = 25   = 0    + 25  
trivial      : 223  = 223  = 0    + 223 
julia        : 273  = 273  = 129  + 144 
r            : 2093 = 2093 = 1991 + 102 
guile        : 41   = 41   = 14   + 27  
elm          : 29   = 29   = 24   + 5   
cargo        : 3342 = 3342 = 195  + 3147
ruby         : 475  = 475  = 80   + 395 
rebar        : 19   = 19   = 11   + 8   
scons        : 14   = 14   = 0    + 14  
font         : 85   = 85   = 61   + 24  
rakudo       : 21   = 21   = 10   + 11  
gnu          : 4241 = 4241 = 812  + 3429
asdf/ecl     : 637  = 637  = 434  + 203 
asdf/sbcl    : 678  = 678  = 453  + 225 
ant          : 462  = 462  = 1    + 461 
meson        : 448  = 448  = 88   + 360 
texlive      : 143  = 143  = 0    + 143 
python       : 2630 = 2630 = 803  + 1827
--8<---------------cut here---------------end--------------->8---


[-- Attachment #2: script.scm --]
[-- Type: text/plain, Size: 1688 bytes --]

(use-modules (guix)
             (gnu)
             (ice-9 match))


(define table (make-hash-table))

(fold-packages (lambda (package result)
                 (let ((bs  (build-system-name
                             (package-build-system package)))
                       (arg (package-arguments    package)))
                   (match (hash-ref result bs)
                     ((tot wo wi)
                      (if (null? arg)
                          (hash-set! result bs (list
                                                (1+ tot)
                                                (1+ wo)
                                                wi))
                          (hash-set! result bs (list
                                                (1+ tot)
                                                wo
                                                (1+ wi)))))
                     (#f (if (null? arg)
                             (hash-set! result bs (list 1 1 0))
                             (hash-set! result bs (list 1 0 1))))
                     (_ (format #t "Error: ~s~%" (package-name package))))
                   result))
              table)


(define fmt "~13s: ~4s = ~4s = ~4s + ~4s~%")
(format #t fmt
        'key 'tot 'tot 'no-arguments 'arguments)
(hash-for-each-handle (lambda (kv)
                        (match kv
                          ((key . value)
                           (match value
                             ((tot wo wi)
                              (format #t fmt
                                      key
                                      (+ wo wi)
                                      tot wo wi))))))
                      table)

Re: guix melpa mirror!

[Mekeor Melire]

2022-11-16 / 07:09 / jgart@dismail.de:

> Have people seen this yet?
> https://github.com/babariviere/guix-emacs

yes, i used this for a while (as you might have seen in the list of
closed issues). but i regularly ran into errors and failures. on the
other hand, the author reacted quite quickly and fixed the mistakes.
still, personally, i didn't find it stable enough to use.

(M)ELPA package metadata accuracy

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> For example, let count the number of packages that are tweaking their
> ’arguments’ fields (from ’#:tests? #f’ to complex phases modifications).
> This is far from being a perfect metrics but it is a rough indication
> about upstream quality: if they provide clean package respecting their
> build system or if the package requires Guix adjustments.
>
> Well, I get:
>
>       r            : 2093 = 2093 = 1991 + 102 
>
> which is good (only ~5% require ’arguments’ tweaks), but
>
>       python       : 2630 = 2630 = 803  + 1827
>
> is bad (only ~31% do not require an ’arguments’ tweak).
>
> About Emacs, it reads,
>
>       emacs        : 1222 = 1222 = 874  + 348 


So that’d be 72% accurate package metadata for (M)ELPA, not too bad!

I tried to estimate repository package data accuracy for my PackagingCon
talk last year in a sophisticated way (perhaps too sophisticated):

  https://git.savannah.gnu.org/cgit/guix/maintenance.git/plain/talks/packaging-con-2021/grail/talk.20211110.pdf
  (slide 53)

  https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00297.html

I think we should identify common sources of inaccuracy in package
metadata and talk with repo maintainers to improve on that.  For MELPA,
it shouldn’t be hard to get in touch so there’s a real opportunity here.

Thanks,
Ludo’.

Re: (M)ELPA package metadata accuracy

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 2334 bytes --]

Hi Ludo,

On Mon, 21 Nov 2022 at 11:00, Ludovic Courtès <ludo@gnu.org> wrote:

> So that’d be 72% accurate package metadata for (M)ELPA, not too bad!

IMHO, one could extrapolate that if someone applies a similar approach
as Lars for CRAN, then 70-75% of (M)ELPA should build out-of-the box.

BTW, unrelated to the topic but note that I engaged a discussion [1]
with Org folk about using Guix as a brick for quality assurance of Emacs
packages – starting with Org. ;-)  Well, the GNU project should reuse
its own components instead of relying on other distros. :-)

1: <https://list.orgmode.org/orgmode/87sfiiycat.fsf@gmail.com/>


> I tried to estimate repository package data accuracy for my PackagingCon
> talk last year in a sophisticated way (perhaps too sophisticated):
>
>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/plain/talks/packaging-con-2021/grail/talk.20211110.pdf
>   (slide 53)
>
>   https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00297.html

IIRC, you mainly investigated the accuracy of the importers…

> I think we should identify common sources of inaccuracy in package
> metadata

…and indeed maybe a systematic analysis of the modifications of the
default emacs-build-system could help in 2 directions:

 1. maybe draw some heuristics to improve the importer accuracy,

>              talk with repo maintainers to improve on that.

and 2. report upstream if some patterns appear.


Well, for packages using ’emacs-build-system’, only comparing ’keyword’
arguments, it reads:

        emacs        : 1234 = 1234 = 878  + 356
            ("phases" . 213)
            ("tests?" . 144)
            ("test-command" . 127)
            ("include" . 87)
            ("emacs" . 25)
            ("exclude" . 20)
            ("modules" . 7)
            ("imported-modules" . 4)
            ("parallel-tests?" . 1) 

Considering this 356 packages, 144 modifies the keyword #:tests?.  Note
that ’#:tests? #t’ is counted in these 144 and it reads,

    $ ag 'tests\? #t' gnu/packages/emacs-xyz.scm | wc -l
    117

Ah!  It requires some investigations. :-)

Indeed, it could be worth to identify common sources of the extra
modifications we are doing compared to the default emacs-build-system.


Cheers,
simon


[-- Attachment #2: script --]
[-- Type: text/plain, Size: 2831 bytes --]

(use-modules (guix)
             (gnu)
             (srfi srfi-1)
             (ice-9 match))


(define %table (make-hash-table))

(define (keyword-as-string arg)
  (map (compose symbol->string keyword->symbol)
       (filter keyword? arg)))

(define (count-items lst)
  (if (null? lst)
      '()
      (let ((sorted (sort lst string<=?)))
        (sort
         (fold (lambda (keyword result)
                 (match result
                   ((head tail ...)
                    (match head
                      ((k . v)
                       (if (string= k keyword)
                           (cons `(,k . ,(+ 1 v)) tail)
                           (cons `(,keyword . 1) result)))
                      (_ (format #t "Error:"))))
                   (_ (format #t "Error:"))))
               (list `(,(car sorted) . 1))
               (cdr sorted))
         (lambda (x1 x2)
           (match x1
             ((k1 . v1)
              (match x2
                ((k2 . v2)
                 (> v1 v2))))))))))


(fold-packages (lambda (package result)
                 (let ((bs  (build-system-name
                             (package-build-system package)))
                       (arg (package-arguments     package)))
                   (match (hash-ref result bs)
                     ((tot wo wi args)
                      (if (null? arg)
                          (hash-set! result bs (list
                                                (1+ tot)
                                                (1+ wo)
                                                wi args))
                          (hash-set! result bs (list
                                                (1+ tot)
                                                wo
                                                (1+ wi)
                                                (append (keyword-as-string arg)
                                                        args)))))
                     (#f (if (null? arg)
                             (hash-set! result bs (list 1 1 0 '()))
                             (hash-set! result bs (list 1 0 1 (keyword-as-string arg)))))
                     (_ (format #t "Error: ~s~%" (package-name package))))
                   result))
              %table)


(define fmt "~13s: ~4s = ~4s = ~4s + ~4s  ~{ ~s ~}~%")
(format #t fmt
        'key 'tot 'tot 'no-arguments 'arguments (list 'pattern?))
(hash-for-each-handle (lambda (kv)
                        (match kv
                          ((key . value)
                           (match value
                             ((tot wo wi args)
                              (format #t fmt
                                      key
                                      (+ wo wi)
                                      tot wo wi (count-items args)))))))
                      %table)