bug#46424: ASDF build system sometimes fails to load .asd files properly

bug#46424: ASDF build system sometimes fails to load .asd files properly

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

Yesterday I pushed 3aba721da73fbdc3382cc098c41596d5cfbb29eb
"gnu: sbcl-quri: Update to 20200209", which fixes the tests for sbcl-quri.

I noted:

--8<---------------cut here---------------start------------->8---
       ;; Test system must be loaded before, otherwise tests fail with:
       ;; Component QURI-ASD::QURI-TEST not found, required by #<SYSTEM
       ;; "quri">.
       '(#:asd-systems '("quri-test" "quri")))
--8<---------------cut here---------------end--------------->8---

Our build system should be able to do better, we should not even have to
specify the asd-systems here.  After all, ASDF is smart enough.

Note that:

- If we don't specify the systems, the tests are still run and
  succeed(!), it's only the phase that fails with the aforementioned
  error.

- If we specify the systems in the reverse order, it still fails!

We've got this issue for many SBCL packages.

Any clue?  Guillaume?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

bug#46424: ASDF build system sometimes fails to load .asd files properly

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Yesterday I pushed 3aba721da73fbdc3382cc098c41596d5cfbb29eb
> "gnu: sbcl-quri: Update to 20200209", which fixes the tests for sbcl-quri.
>
> I noted:
>
> --8<---------------cut here---------------start------------->8---
>        ;; Test system must be loaded before, otherwise tests fail with:
>        ;; Component QURI-ASD::QURI-TEST not found, required by #<SYSTEM
>        ;; "quri">.
>        '(#:asd-systems '("quri-test" "quri")))
> --8<---------------cut here---------------end--------------->8---
>
> Our build system should be able to do better, we should not even have to
> specify the asd-systems here.  After all, ASDF is smart enough.
>
> Note that:
>
> - If we don't specify the systems, the tests are still run and
>   succeed(!), it's only the phase that fails with the aforementioned
>   error.
>
> - If we specify the systems in the reverse order, it still fails!
>
> We've got this issue for many SBCL packages.
>
> Any clue?  Guillaume?

Currently we are using the 'asdf:load-asd' function to load system
definitions (see the 'build' function in
'guix/build/asdf-build-system.scm' and the 'compile-systems' function in
'guix/build/lisp-utils.scm').

This function can only load one '.asd' file at a time, which is the
cause of the issue. If we find how ASDF does the "lazy-load" of all the
'.asd' files in a directory tree, and if we use that instead of
'load-asd', I think it would solve the issue and we could get rid of the
'asd-files' parameter in Guix package definitions.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#46424: Use load-systems or load-systems*

[Sharlatan Hellseher]

Hi I've just checked source of ASDF and it looks like there is a
native option to load multiple systems at once

> --8<---------------cut here---------------start------------->8--->
  (defun load-systems* (systems &rest keys)
    "Loading multiple systems at once."
    (dolist (s systems) (apply 'load-system s keys)))

  (defun load-systems (&rest systems)
    "Loading multiple systems at once."
    (load-systems* systems))
--8<---------------cut here---------------end--------------->8---

https://gitlab.common-lisp.net/asdf/asdf/-/blob/master/operate.lisp#L155

-- 
… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

bug#46424: Use load-systems or load-systems*

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

In the case of quri, it looks like the problem comes from the
"(asdf:clear-system c)" call at the end of "quri-test.asd". When
removing it, the build succeeds even without the custom 'asd-systems'
argument in the Guix package definition.

So it looks like the lazy loading of system definitions is working in
asdf-build-system after all. I guess it's the default behaviour of
'load-asd'.

However I don't know why this 'clear-system' call messes up the ASDF
configuration in asdf-build-system, but it doesn't seem to cause
problems in Quicklisp...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#46424: Use load-systems or load-systems*

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

Interesting.
This reminds me of a puzzling issue I recently had while working on the
Nyxt .asd:

- (asdf:load-asd "/path/to/nyxt.asd")
- (asdf:make :nyxt/quicklisp)

The nyxt/quicklisp system would call ql:update-dist and then would fail,
saying it cannot find "nyxt/quicklisp" (which it is running!)
because ql:update-dist would lead ASDF to somehow forget about the .asd.

Adding (asdf:load-asd "/path/to/nyxt.asd") right after ql:update-dist
fixes it.

I've reworked the whole thing different on Nyxt's master branch, so this
is no longer an issue.

Maybe there is a hint lurking in there...

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

bug#46424: ASDF build system sometimes fails to load .asd files properly

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 83 bytes --]

This has been fixed with commit
6b5ef03a2582ab23228478018fd356e17db1daea.
Closing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]