[bug#59592] Keep asd-systems for cl-* packages

[bug#59592] Keep asd-systems for cl-* packages

[Charles via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 112 bytes --]

Greetings Guix patches!

Relevant thread: https://mail.gnu.org/archive/html/guix-devel/2022-11/msg00153.html

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-asdf-build-system-Keep-some-package-arguments-f.patch --]
[-- Type: text/x-patch; name=0001-guix-asdf-build-system-Keep-some-package-arguments-f.patch, Size: 1148 bytes --]

From 990cb50b8ba49c2bf1787abdffa54318d285624c Mon Sep 17 00:00:00 2001
From: Charles <charles.b.jackson@protonmail.com>
Date: Thu, 24 Nov 2022 11:44:12 -0600
Subject: [PATCH] guix: asdf-build-system: Keep some package-arguments for cl-*
 packages.

Specifically asd-system and asd-test-systems. Before theses were striped
out. Now they stay there as meta data not used by the build-system.

* guix/build-system/asdf.scm (package-with-build-system):
Remove #:asd-systems #:asd-test-systems from striped arguments.
---
 guix/build-system/asdf.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/guix/build-system/asdf.scm b/guix/build-system/asdf.scm
index 74a3e47da1..2985f7f786 100644
--- a/guix/build-system/asdf.scm
+++ b/guix/build-system/asdf.scm
@@ -203,7 +203,7 @@ (define (new-inputs inputs-getter)
       (define base-arguments
         (if target-is-source?
             (strip-keyword-arguments
-             '(#:tests? #:lisp #:asd-systems #:asd-test-systems #:asd-operation)
+             '(#:tests? #:lisp #:asd-operation)
              (package-arguments pkg))
             (package-arguments pkg)))
 
-- 
2.38.1

[bug#59592] Keep asd-systems for cl-* packages

[Charles via Guix-patches via]

Hey Guillaume. Do you have an opinion on this. I'm asking since you were the only one who responded to the initial thread. Thanks for taking a look by the way.

[bug#59592] Keep asd-systems for cl-* packages

[Guillaume Le Vaillant]

[-- Attachment #1.1: Type: text/plain, Size: 1540 bytes --]

Hi,

The 'asd-systems' field contains the minimal subset of the ASDF systems
defined by a library that, when compiled, compiles all the systems of
the library.

For example, the ironclad package has many subsystems:
  "ironclad/ciphers"
  "ironclad/digests"
  "ironclad/cipher/aes"
  "ironclad/cipher/threefish"
  "ironclad/digest/sha256"
  "ironclad/public-key/ed25519"
  ...
but the 'asd-systems' field only contains "ironclad".

The mcclim package also has many subsystems:
  "automaton"
  "clim"
  "clim-core"
  "clim-core/frames"
  "clim-core/system"
  "drei-mcclim"
  "esa-mcclim"
  "mcclim-backend-common"
  "mcclim-fonts"
  "slim"
  ...
but the 'asd-systems' field only contains the top level ones, "mcclim",
"clim-examples" and "clim-debugger".

So you can't rely on the contents of 'asd-systems' to have a complete
list of the systems defined by a library.

You could get the full list by searching the for 'defsystem' forms in
the sources directly, with something like:

--8<---------------cut here---------------start------------->8---
find sources -name "*.asd" -exec grep "(defsystem" {} \; | cut -d ' ' -f 2
--8<---------------cut here---------------end--------------->8---

But it would only work for systems not using the package-inferred-system
feature of ASDF.


In my files, I found a function (I don't remember where I got it from)
that can get all the dependencies of a loaded system, even the ones
using package-inferred-system. Maybe it could be modified to limit the
results to a specific library's source tree...


[-- Attachment #1.2: get-dependencies.lisp --]
[-- Type: application/octet-stream, Size: 2904 bytes --]

(asdf!load-system :fset)

(defun get-dependencies (system)
  "Returns a set with all dependencies of a given system.
   System should be loaded first."
  (labels ((normalize (name)
             (etypecase name
               (string (string-downcase name))
               (symbol (normalize (symbol-name name)))
               (list
                (let ((dep-type (first name))
                      (supported-dep-types (list :version :feature :require)))
                  (unless (member dep-type
                                  supported-dep-types)
                    (error "This component \"~A\" should have first element from this list: ~A."
                           name
                           supported-dep-types))
                  
                  (normalize
                   (case dep-type
                     (:version (second name))
                     (:feature (third name))
                     (:require (second name)))))))))
    
    (let ((processed (fset:set))
          (queue (fset:set (normalize system))))
      
      (do ((current-name (fset:arb queue)
                         (fset:arb queue)))
          ((null current-name)
           ;; return result
           processed)

        ;; Remove current name from the queue
        (setf queue
              (fset:less queue current-name))
        ;; And put it into the "processed" pool
        (setf processed
              (fset:with processed current-name))
        
        ;; And add it's dependencies which aren't processed or in the queue already
        ;; Sometimes system can't be found because itself depends on some feature,
        ;; for example, you can specify dependency as a list:
        ;; (:FEATURE :SBCL (:REQUIRE :SB-INTROSPECT))
        ;; and it will be loaded only on SBCL.
        ;; When we are collecting dependencies on another implementation,
        ;; we don't want to fail with an error because ASDF is unable to find
        ;; such dependencies
        (let* ((system (ignore-errors
                        (asdf:find-system current-name)))
               (deps (when system
                       (asdf:component-sideway-dependencies system))))
          (dolist (dep deps)
            (let ((normalized-dep (normalize dep)))
              (unless (or (fset:lookup processed normalized-dep)
                          (fset:lookup queue normalized-dep))
                (setf queue
                      (fset:with queue normalized-dep)))))))

      (values processed))))

#|
DEPENDENCIES> (asdf:load-system :clinch)
DEPENDENCIES> (get-dependencies :clinch)
#{
  "cffi"
  "sdl2"
  "uiop"
  "babel"
  "swank"
  "clinch"
  "cl-glut"
  "cl-json"
  "cl-ppcre"
  "rtg-math"
  "cl-opengl"
  "cl-plus-c"
  "alexandria"
  "cl-autowrap"
  "glsl-symbols"
  "defpackage-plus"
  "trivial-garbage"
  "trivial-timeout"
  "bordeaux-threads"
  "trivial-channels"
  "trivial-features" }
|#

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

[bug#59592] (No Subject)

[Charles via Guix-patches via]

I didn't know that package-inferred-systems also coupled asdf systems in addition to packages and files. I'm not a fan of package-inferred-system if you couldn't tell.

What are your thoughts on a patch that would add all systems to #:asd-systems? I could use your program along with some manual filtering to filter it out. It is ideal to have the knowledge of all asd-systems of a project before loading it.

[bug#59592] (No Subject)

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

Charles <Charles.b.jackson@proton.me> skribis:

> I didn't know that package-inferred-systems also coupled asdf systems in addition to packages and files. I'm not a fan of package-inferred-system if you couldn't tell.
>
> What are your thoughts on a patch that would add all systems to #:asd-systems?
> I could use your program along with some manual filtering to filter it out. It
> is ideal to have the knowledge of all asd-systems of a project before loading
> it.

Adding everything to 'asd-systems' would be tedious. I think Guix
package definitions are not the right place to keep the complete list of
available ASDF systems (just like we don't keep the list of ".so"
C libraries in package definitions).

There's a file search functionality that is being developped (there's
some info about it in the guix-devel mailing list I think). Maybe it
could be used as a base for a script searching for ASDF systems.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

[bug#59592] (No Subject)

[Charles via Guix-patches via]

> Adding everything to 'asd-systems' would be tedious. 

We can agree on that point.

> I think Guix package definitions are not the right place to keep the 
> complete list of available ASDF systems (just like we don't keep the 
> list of ".so" C libraries in package definitions).

But where is a better place? I was planning to open up the conversation 
about doing just that for C libraries 😁 (and executable binaries, and
python packages, and rust crates, and so on).
 
> There's a file search functionality that is being developped (there's
> some info about it in the guix-devel mailing list I think). Maybe it
> could be used as a base for a script searching for ASDF systems.

This one: https://lists.gnu.org/archive/html/guix-devel/2022-12/msg00234.html
Subject: "File Search".

I have a couple gripes with that approach:

1. It only searches for files (using a file database). This makes it hard to 
differentiate searching between different types of files. This also makes it 
hard to search for something like an asdf system which might not be associated 
with a file on a 1 to 1 basis.

2. It only index the searchable stuff after building the packages. This raises
a whole lot of issues mentioned in that thread already (networking, privacy, etc.).
My approach would make the information available at the time of guix pull in the
package definition.