ASDF Builder (Common Lisp) & "package-inferred-system" Packages

ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Katherine Cox-Buday]

Hey all.

I think the common-lisp build system might be broken for
"package-inferred-system" style common-lisp packages. The build system
appears to find the dependencies, but then replace[1] "invalid" (within
the context of the store) characters with `-`. I think this breaks asdf
as it then cannot find the package. "package-inferred-system" packages
almost always contain a `/`[2] which is flagged as an invalid
character.

Do we even need to be doing scrubbing here since it's within the context
of ASDF, not the store?

Can anyone confirm, or does anyone have any thoughts? This style of
Common Lisp package will probably increase moving forward, so we should
probably address this sooner than later.

[1] - http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/lisp-utils.scm#n165
[2] - https://www.common-lisp.net/project/asdf/asdf.html#index-Package-inferred-systems

-- 
Katherine

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Pierre Neidhardt]

Hi Katherine,

I've packaged a lot of Lisp packages, none of which seem to suffer from a naming
issue.
Do you have an example in mind?

Which string replacement are you referring to?  NORMALIZE-STRING?

I'm not completely sure, but I think that in practice packages can always
specify the right ASD-FILE or SYSTEM.  There could be something missing though.

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Katherine Cox-Buday]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> I've packaged a lot of Lisp packages, none of which seem to suffer
> from a naming issue.

This is a newer style of ASDF system. I don't think any of the things we
have packaged use this style (yet).

> Do you have an example in mind?

I have 41 packages almost ready to go, but the only one I'm currently
packaging that uses the "package-inferred-system" style is Ningle[1].

> Which string replacement are you referring to?  NORMALIZE-STRING?

Yes. I'm wondering why we even need to call `normalize-string' here
since this is generating the string ASDF will use to search for a
package, and ASDF is capable of handling `/` characters in its package
names. I think we may be conflating the need for the store to remove `/`
characters with ASDF's needs, but I'm not completely sure.

> I'm not completely sure, but I think that in practice packages can
> always specify the right ASD-FILE or SYSTEM. There could be something
> missing though.

This is not about specifying the ASD file; it is regarding how a system
is defined within an ASD file. Please read this[2] link for more
information. A single ASD file will define a package (note, not system)
per file. The packages will all have the `/` character embedded in the
name since it is correlating files in the system's path with packages.

I cannot find a way to tell the runtime that the renamed package (in
this case "ningle-main") is an alias for the real package
("ningle/main"), but it is very possible I'm missing something obvious.

[1] - https://github.com/fukamachi/ningle/blob/master/ningle.asd
[2] - https://www.common-lisp.net/project/asdf/asdf.html#index-Package-inferred-systems

-- 
Katherine

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Katherine Cox-Buday]

I've done some more digging, and I have a better idea of why this is
failing.

Running `(asdf:operate 'asdf:compile-bundle-op :foo)` on a
`package-inferred-system` produces several `.fasl` files -- one for each
sub-package instead of the usual `foo--system.fasl` file. The
`cleanup-files` phase then deletes[1] any `.fasl` files which do not have a
`--system.fasl` suffix.

I tried working around this by manually concatenating all the `.fasl`
files together using `uiop:combine-fasls` which I believe works. I also
had to add a new build phase to take place after the `create-asd-file`
phase to do a string replace to change dependencies back to the
`foo/bar` style instead of `foo-bar`.

I still believe the build system should handle `package-inferred-system`
style CL packages properly so that maintainers don't have to do this
kind of tweaking for every package (the number of which I believe will
steadily increase).

I'm trying to confirm that this whole things works, but I believe I may
be hitting another bug with CL being able to find dependencies. It seems
as though even if the dependenices are listed in the generated `.asd`
file, and my Guix SBCL can find that `.asd` and attempt to load its
system, SBCL cannot find the dependencies.

I'm trying to determine why this might be, but it's currently blocking
me from proposing a patch which adds 44 CL packages.

My current theory is that we perform a `setf` on the
`*source-registry-parameter*` variable, but the ASDF manual
specifically, and strongly, says[2] that this is meant to be a read-only
variable. Maybe this is not working as intended?

I'm also going through some steps to establish a clean-room to make sure
it's nothing with my setup.

If anyone feels they can point me in helpful directions, that would be
very much appreciated :)

[1] - http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/asdf-build-system.scm#n193
[2] - https://www.common-lisp.net/project/asdf/asdf.html#g_t_002asource_002dregistry_002dparameter_002a-variable

-- 
Katherine

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Andy Patterson]

Hey Katherine,

On Sat, 12 Jan 2019 14:24:38 -0600
Katherine Cox-Buday <cox.katherine.e@gmail.com> wrote:

> I've done some more digging, and I have a better idea of why this is
> failing.
> 
> Running `(asdf:operate 'asdf:compile-bundle-op :foo)` on a
> `package-inferred-system` produces several `.fasl` files -- one for
> each sub-package instead of the usual `foo--system.fasl` file. The
> `cleanup-files` phase then deletes[1] any `.fasl` files which do not
> have a `--system.fasl` suffix.
> 

This phase has always been a bit of a hack.  There are already some
other packages where asdf doesn't produce an output file which matches
the expected name, so we had to change that.  I'd be in favour of
removing the phase wholesale until we can create one which works
better.  Ultimately we should be querying asdf for the output files
that it produced, and allowing the phase to be switched off with an
argument.

> I tried working around this by manually concatenating all the `.fasl`
> files together using `uiop:combine-fasls` which I believe works. I
> also had to add a new build phase to take place after the
> `create-asd-file` phase to do a string replace to change dependencies
> back to the `foo/bar` style instead of `foo-bar`.
> 

We've done something similar (the opposite) with the slynk package so
it's not unheard of.  If there's a better way to handle it though, we
should give it a go.

> I still believe the build system should handle
> `package-inferred-system` style CL packages properly so that
> maintainers don't have to do this kind of tweaking for every package
> (the number of which I believe will steadily increase).
> 

Definitely agree here.  We currently don't have overly many packages
relying on the build system so now would be a great time to fix it as
much as possible.

> I'm trying to confirm that this whole things works, but I believe I
> may be hitting another bug with CL being able to find dependencies.
> It seems as though even if the dependenices are listed in the
> generated `.asd` file, and my Guix SBCL can find that `.asd` and
> attempt to load its system, SBCL cannot find the dependencies.
> 
> I'm trying to determine why this might be, but it's currently blocking
> me from proposing a patch which adds 44 CL packages.
> 

Hmm, I'd need to see more information here to be able to offer anything
useful.  I think I've ran into similar problems in the past but I don't
remember what the solutions were.

> My current theory is that we perform a `setf` on the
> `*source-registry-parameter*` variable, but the ASDF manual
> specifically, and strongly, says[2] that this is meant to be a
> read-only variable. Maybe this is not working as intended?
> 

It's possible - this was the best way that I found at the time to
associate a system with a file directly.  AFAIK there's no way to ask
asdf to load a system from a file directly.  Any suggestions here are
welcome.

> [...]

Thanks,

--
Andy

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Katherine Cox-Buday]

Andy Patterson <ajpatter@uwaterloo.ca> writes:

> Hey Katherine,

Hey Andy! I'm so glad you're chiming in. Thanks for providing this build
system!

> On Sat, 12 Jan 2019 14:24:38 -0600 Katherine Cox-Buday
> <cox.katherine.e@gmail.com> wrote:
>
>> I've done some more digging, and I have a better idea of why this is
>> failing.
>> 
>> Running `(asdf:operate 'asdf:compile-bundle-op :foo)` on a
>> `package-inferred-system` produces several `.fasl` files -- one for
>> each sub-package instead of the usual `foo--system.fasl` file. The
>> `cleanup-files` phase then deletes[1] any `.fasl` files which do not
>> have a `--system.fasl` suffix.
>> 
>
> This phase has always been a bit of a hack. There are already some
> other packages where asdf doesn't produce an output file which matches
> the expected name, so we had to change that. I'd be in favour of
> removing the phase wholesale until we can create one which works
> better. Ultimately we should be querying asdf for the output files
> that it produced, and allowing the phase to be switched off with an
> argument.

ASDF also has the ability to search trees, so we may choose to abandon
the concept of a single fasl file altogether and instead create
`/lib/foo/*.fasl` folders. We could then let the fasl files ASDF
produces lay about the system as generated :) I don't yet have a
fully-formed opinion about this.

>> I tried working around this by manually concatenating all the
>> `.fasl` files together using `uiop:combine-fasls` which I believe
>> works. I also had to add a new build phase to take place after the
>> `create-asd-file` phase to do a string replace to change
>> dependencies back to the `foo/bar` style instead of `foo-bar`.
>> 
>
> We've done something similar (the opposite) with the slynk package so
> it's not unheard of. If there's a better way to handle it though, we
> should give it a go.
>
>> I still believe the build system should handle
>> `package-inferred-system` style CL packages properly so that
>> maintainers don't have to do this kind of tweaking for every package
>> (the number of which I believe will steadily increase).
>> 
>
> Definitely agree here. We currently don't have overly many packages
> relying on the build system so now would be a great time to fix it as
> much as possible.

I haven't yet gotten `:ningle` (my test package-inferred-system) to
work. It complains that it can't find `ningle/main`, and I don't know
why as that appears to be in the UIOP bundled fasl.

As we are approaching Guix 1.0, this would also be a good time to shore
this up so that CL users trying Guix for the first time have a pleasant
experience.

>> I'm trying to confirm that this whole things works, but I believe I
>> may be hitting another bug with CL being able to find dependencies.
>> It seems as though even if the dependenices are listed in the
>> generated `.asd` file, and my Guix SBCL can find that `.asd` and
>> attempt to load its system, SBCL cannot find the dependencies.
>> 
>> I'm trying to determine why this might be, but it's currently
>> blocking me from proposing a patch which adds 44 CL packages.
>> 
>
> Hmm, I'd need to see more information here to be able to offer
> anything useful. I think I've ran into similar problems in the past
> but I don't remember what the solutions were.

It was an issue on my end. In my
${HOME}/.config/common-lisp/source-registry.conf.d/asdf.conf file, I
had `(:tree "~/.guix-profile/share/common-lisp/sbcl-source/")` defined
instead of `(:tree "~/.guix-profile/lib/sbcl")`. This allowed ASDF to
find the source definitions, but not the `.asd` files including the
pointers to the packages' dependencies. Changing this completely solved
the dependency issue, although I think we ought to find a different way
to point to dependencies as ASDF may break us at some point.

>> My current theory is that we perform a `setf` on the
>> `*source-registry-parameter*` variable, but the ASDF manual
>> specifically, and strongly, says[2] that this is meant to be a
>> read-only variable. Maybe this is not working as intended?
>> 
>
> It's possible - this was the best way that I found at the time to
> associate a system with a file directly. AFAIK there's no way to ask
> asdf to load a system from a file directly. Any suggestions here are
> welcome.

I will continue to do some digging to try and form some kind of cogent
argument for an approach. To start, I need to get ningle to even work
when packaged!

-- 
Katherine

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3057 bytes --]

Hi!

It's been a while, and now that I'm on the verge of releasing Next 1.3
and a dependency is a package-inferred-system, I thought it would be the
perfect time to dig out this issue! :)

So I'm trying to package cl-dbus (https://github.com/death/dbus).
You'll find the recipes here: https://gitlab.com/atlas-engineer/next-guix-channel

I can package it in several ways, but the result does not work for the
reasons mentioned in this topic.

I thought of the following trick: what about just dumping the full source
into "/lib/sbcl"?  Wouldn't that work?
Do we really need the .fasl files?

When I try to build Next with dbus packaged using this last trick, it
actually gets quite far, until it complains about circular dependencies:

--8<---------------cut here---------------start------------->8---
Unhandled ASDF/ACTION:CIRCULAR-DEPENDENCY in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                                       {10005385B3}>:
  Circular dependency:
     ((#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus/all">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "dbus/all" "lisp">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "dbus/all" "lisp">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus/all">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus/convenience">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "dbus/convenience" "lisp">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "dbus/convenience" "lisp">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus/convenience">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus/publish">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "dbus/publish" "lisp">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "dbus/publish" "lisp">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/PACKAGE-INFERRED-SYSTEM:PACKAGE-INFERRED-SYSTEM "dbus/publish">))
--8<---------------cut here---------------end--------------->8---

I've asked for help on ASDF issue tracker:

https://gitlab.common-lisp.net/asdf/asdf/issues/10


By the way, there they confirmed that using `*source-registry*` is the
right way to go.  Note that this is _not_ the same variable as
`*source-registry-parameter*` that Katherine mentioned.

So we are good on that front.

Thoughts?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

I just figured out something: it works if I use the source-only
(cl-dbus) version of the package.  What's broken is the sbcl-dbus (and I
assume ecl-dbus).  But that's not big deal, in fact it's perfectly fine
to ship the source-only version of a single library, it blends in with
the rest since all(?) Common Lisp compilers compile on demand.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2932 bytes --]

The "asdf-build-system/source" workaround seems to have a limitation
though: it does not include its input recursively.

For instance, if I try to build Next that depends on cl-dbus, it will
complain about missing dependencies for cl-dbus.
Adding

--8<---------------cut here---------------start------------->8---
       ,@(package-inputs cl-dbus)
--8<---------------cut here---------------end--------------->8---

to the list of dependencies fixes it, but that's not pretty.

I believe that this is a general issue with asdf-build-system/source
(could be wrong).
If I look at the environment-variables file when building Next, it has

--8<---------------cut here---------------start------------->8---
export XDG_DATA_DIRS=\
"/gnu/store/2fkvzws2vqhd7zq265n2a3rjkdmc0i6v-sbcl-1.5.1/share\
:/gnu/store/qc74s7g1z3pm11d0kfka9pkc5af11pa6-sbcl-prove-asdf-1.0.0-1.4f9122b/share\
:/gnu/store/aglych5kld05hip5j6wn5b8yx66hpagd-next-gtk-webkit-0.0.0/share\
:/gnu/store/6l1qcw9731psvp4qp28dvq9j2g1nbnmm-sbcl-trivial-features-0.8/share\
:/gnu/store/g82q0i0wah4d2qh012yi793rvg526s59-sbcl-trivial-garbage-0.21/share\
:/gnu/store/x8k2la7c69dfr62vrgijajp81hybd79s-sbcl-alexandria-0.0.0-1.926a066/share\
:/gnu/store/jdnsak2ylxshg27cxr9fh9gbx1zpwzar-sbcl-bordeaux-threads-0.8.6-1.5dce49f/share\
:/gnu/store/52qzpiy63l6yy9n0icryvf317hiwiiyq-sbcl-cl-css-0.1-1.8fe654c/share\
:/gnu/store/sbhkpri9avslvsmzv0fahswbp9g4igy3-sbcl-cl-json-0.5-1.6dfebb9/share\
:/gnu/store/44dsabdh9fn1j4m9gcll5lqmrnycss9z-sbcl-cl-markup-0.1-1.e0eb7de/share\
:/gnu/store/99ynic4nmhq4mxbrj1rx66lvbw3gki01-sbcl-cl-ppcre-2.0.11/share\
:/gnu/store/ksxf4v32xcmhw3vnrhq8rkyvzwgfx7vk-sbcl-cl-ppcre-unicode-2.0.11/share\
:/gnu/store/ry9pmrqyxgyva7l5747g7ip6mgwkksys-sbcl-cl-string-match-0-1.5048480/share\
:/gnu/store/1gsm5f75rv4isdylbzrkvycw2gh2969h-sbcl-cl-strings-0.0.0-1.c5c5cba/share\
:/gnu/store/zpv26q5jgsr7hbgp3rb3nxhkcl1g6r5z-sbcl-closer-mop-1.0.0-1.fac29ce/share\
:/gnu/store/k1m8m4wxkvqx354h0f48650a57c4ivgk-cl-dbus-20190408-1.24b452d/share\
:/gnu/store/axfsl7653s96cicv7rx1rm26amxjyalr-sbcl-dexador-0.9.10-1.a2714d1/share\
:/gnu/store/3wbyra9x6qjlfp2p3ijxpnlz5pn33xpw-sbcl-ironclad-0.42/share\
...
--8<---------------cut here---------------end--------------->8---

I think the XDG_DATA_DIRS is set automatically from the set of direct inputs,
but it's does not include the recursive inputs.

For asdf-build-system/source to work properly, we would need to force
the input.

Actually, asdf.scm has this comment:

--8<---------------cut here---------------start------------->8---
      ;; Special considerations for source packages: CL inputs become
      ;; propagated, and un-handled arguments are removed.
--8<---------------cut here---------------end--------------->8---

But I think that propagating the inputs might not suffice to build other
packages.

Thoughts?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Katherine Cox-Buday]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Hi!
>
> It's been a while, and now that I'm on the verge of releasing Next 1.3
> and a dependency is a package-inferred-system, I thought it would be the
> perfect time to dig out this issue! :)
>
> So I'm trying to package cl-dbus (https://github.com/death/dbus).
> You'll find the recipes here:
> https://gitlab.com/atlas-engineer/next-guix-channel
>
> I can package it in several ways, but the result does not work for the
> reasons mentioned in this topic.

Hello again, Pierre! I apologize for being silent on this for so long.
I'm juggling a few things at the moment and will return to Guix commits
as soon as I can.

Regarding package inferred systems: I would really like to be able to
package them into a single Guix package as we do any other library, but
I think that will require some hacking on the ASDF build system. If I'm
remembering correctly, we would have to:

- Stop rolling all code into system-fasls
- Output each file's equivalent fasl to a common directory for the
  system
- Set a search path in the profile to recursively search the SBCL fasl
  output tree for systems.

In the interim, an idea I had was to borrow the "base package" concept I
came across when packaging other, non package-inferred, systems. I
haven't tried it yet, but I think it should work, even if it sounds
tedious to me.

Sticking with the dbus example, it looks like we have the following
situation:

- dbus
  - dbus/all
    - dbus/utils
    - dbus/protocols
    - ...

I think what we could do is define a `dbus-base` package which is
hidden, and then start defining "sub-packages" which simply inherit from
the base package, i.e.:

- dbus-base
  - dbus-utils (inherit from dbus-base)
  - dbus-protocols (inherit from dbus-base)
  - ...
  - dbus-all (inherit from dbus-base; require dbus-utils,
    dbus-protocols, etc.)

In essence we would be manually defining the dependency tree that ASDF
is inferring. We could probably lean on Common Lisp and ASDF itself to
generate these packages within a hypothetical importer.

But as I said, this seems quite tedious. We really should make our
Common Lisp packaging smarter.

-- 
Katherine

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2464 bytes --]

Hi Katherine!

One of the issue that we have with the current build system is that it
does not find the dependencies properly for the inferred packages.

Robert from the ASDF team suggested a solution:

			https://gitlab.common-lisp.net/asdf/asdf/issues/10

I think if we use his more general function, we can properly populate
the *source-registry*.  So that would be a solution to the first of our
problems, that of the missing inputs.

The main other issue is the one you mention: we need to compile the
.fasl files right.

> - Stop rolling all code into system-fasls
> - Output each file's equivalent fasl to a common directory for the
>   system
> - Set a search path in the profile to recursively search the SBCL fasl
>   output tree for systems.

This sounds reasonable.  Also you mentioned above that concatenating the
generated fasl files into a single one manually worked for you.  If it
works, then why not!


> In the interim, an idea I had was to borrow the "base package" concept I
> came across when packaging other, non package-inferred, systems. I
> haven't tried it yet, but I think it should work, even if it sounds
> tedious to me.
>
> Sticking with the dbus example, it looks like we have the following
> situation:
>
> - dbus
>   - dbus/all
>     - dbus/utils
>     - dbus/protocols
>     - ...
>
> I think what we could do is define a `dbus-base` package which is
> hidden, and then start defining "sub-packages" which simply inherit from
> the base package, i.e.:
>
> - dbus-base
>   - dbus-utils (inherit from dbus-base)
>   - dbus-protocols (inherit from dbus-base)
>   - ...
>   - dbus-all (inherit from dbus-base; require dbus-utils,
>     dbus-protocols, etc.)
>
> In essence we would be manually defining the dependency tree that ASDF
> is inferring. We could probably lean on Common Lisp and ASDF itself to
> generate these packages within a hypothetical importer.

I don't think that would work, because the inferred packages can have
circular dependencies, e.g. dbus-utils depends on dbus-protocols which
depends on dbus-utils (correct me if I'm wrong).
But this is not allowed in Guix (the dependency graph is acyclic).

> But as I said, this seems quite tedious. We really should make our
> Common Lisp packaging smarter.

I believe your above solution could work.  Wanna give it a shot?

Andy, what do you think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

Correction: I was wrong, it seems that inferred packages don't play well
with circular dependencies.  So then the "base package" approach might
work, but still as Katherine said it would be more ideal to not have to
resort to it and package everything into a single package.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Katherine Cox-Buday]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Hi Katherine!
>
> One of the issue that we have with the current build system is that it
> does not find the dependencies properly for the inferred packages.
>
> Robert from the ASDF team suggested a solution:
>
> 			https://gitlab.common-lisp.net/asdf/asdf/issues/10
>
> I think if we use his more general function, we can properly populate
> the *source-registry*.  So that would be a solution to the first of our
> problems, that of the missing inputs.

Perfect!

> The main other issue is the one you mention: we need to compile the
> .fasl files right.
>
>> - Stop rolling all code into system-fasls
>> - Output each file's equivalent fasl to a common directory for the
>>   system
>> - Set a search path in the profile to recursively search the SBCL fasl
>>   output tree for systems.
>
> This sounds reasonable.  Also you mentioned above that concatenating the
> generated fasl files into a single one manually worked for you.  If it
> works, then why not!

I never got this to work[1]. I think Ningle's systems are being bundled
into a single fasl, but for whatever reason, its subsystems cannot be
found despite that bundled fasl being on the path.

Last I looked at this, I remember being left with the distinct
impression that we would have to stop producing system fasls, and rather
than having one directory of system fasls, we'd have a tree of system
directories, each populated with all the fasls for a system e.g.:

- ~/.guix-profile/lib/sbcl/
  - dbus/
    - dbus-all.fasl
    - dbus-utils.fasl
    - ...

And then find a way to search ~/.guix-profile/lib/sbcl/ recursively. The
ASDF manual says[2] $XDG_CONFIG_DIRS/common-lisp/source-registry.conf
can hold configurations, as well as `CL_SOURCE_REGISTRY`. Maybe Guix
could set one of these in the profile to something like `(:tree
(:home ".guix-profile/lib/sbcl/"))`.

Or we could stick to the `*source-registry*` variable and use a smarter
Common Lisp function to populate it.

> I believe your above solution could work.  Wanna give it a shot?

Currently I have 1 day a week to work on open source things, and I have
been busy with other projects. Those may be wrapping up "soon", and then
I could give it a go, but I don't think you want to wait for me :) I
also don't speak Guile very well, and these larger changes to the build
system would probably take me awhile. It would probably be faster and
help to educate me if I were a reviewer on patches you produce.

I also want to mention that I remember going around in circles on this a
bit, so take everything I say here with a grain of salt. I think
experimentation is needed to find the cleanest implementation!

> Andy, what do you think?

[1]- https://lists.gnu.org/archive/html/guix-devel/2019-01/msg00204.html
[2] - https://www.common-lisp.net/project/asdf/asdf.html#Configurations

-- 
Katherine

Re: ASDF Builder (Common Lisp) & "package-inferred-system" Packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1686 bytes --]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> I never got this to work[1]. I think Ningle's systems are being bundled
> into a single fasl, but for whatever reason, its subsystems cannot be
> found despite that bundled fasl being on the path.

According to Fare in the GitLab discussion, this is to be expected.  It
seems that generally fasls cannot be bundled for inferred packages.

> Last I looked at this, I remember being left with the distinct
> impression that we would have to stop producing system fasls, and rather
> than having one directory of system fasls, we'd have a tree of system
> directories, each populated with all the fasls for a system e.g.:
>
> - ~/.guix-profile/lib/sbcl/
>   - dbus/
>     - dbus-all.fasl
>     - dbus-utils.fasl
>     - ...
>
> And then find a way to search ~/.guix-profile/lib/sbcl/ recursively. The
> ASDF manual says[2] $XDG_CONFIG_DIRS/common-lisp/source-registry.conf
> can hold configurations, as well as `CL_SOURCE_REGISTRY`. Maybe Guix
> could set one of these in the profile to something like `(:tree
> (:home ".guix-profile/lib/sbcl/"))`.
>
> Or we could stick to the `*source-registry*` variable and use a smarter
> Common Lisp function to populate it.

My current understanding is that *source-registry* is the ideal way to
go.  So we would just need to add the /lib/sbcl folder and that should
be enough?
As far as I understand, that's how the ~/.cache/common-lisp fasls are loaded.



This is not too urgent for me anymore since cl-dbus works (source), so I
might only work on this when I have more time in the future.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]