Creating a package using two sources/origins

Creating a package using two sources/origins

[dabbede]

Dear Guix members,

I would like to contribute to the project by creating a package for
linux-xenomai (a real-time extension to the linux kernel, see
https://source.denx.de/Xenomai/xenomai/-/wikis/home).
My issue is that, according to xenomai's build instructions, I am
supposed to use both a mainstream patched kernel (I'm pretty sure that
a deblobbed linux-libre kernel can work too) and a .tar.bz2 source
tree that contains the user-space libraries and binaries but also some
scripts for preparing the kernel before building.

By looking into linux-libre package definition, I see no problem in
downloading and patching the proper kernel source, however I don't
understand how I am supposed to specify in the package definition that
we need an additional source tree in the build process.
What would you suggest? Using snippets or additional build-phases?
Either way, I'm not skilled enough to download a compressed tar and
unpack it in a reachable directory at the build stage. I'm only aware
of the package's "origin" function to accomplish that. What is the
best way to do the same in a snippet or phase?

Thanks for the advice. I need this input to study more the topic ;-)

Regards, Davide

Re: Creating a package using two sources/origins

[(]

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

Heya,

On Mon Dec 12, 2022 at 10:35 PM GMT, dabbede@gmail.com wrote:
> By looking into linux-libre package definition, I see no problem in
> downloading and patching the proper kernel source, however I don't
> understand how I am supposed to specify in the package definition that
> we need an additional source tree in the build process.
> What would you suggest? Using snippets or additional build-phases?
> Either way, I'm not skilled enough to download a compressed tar and
> unpack it in a reachable directory at the build stage. I'm only aware
> of the package's "origin" function to accomplish that. What is the
> best way to do the same in a snippet or phase?

What you need to do is combine ORIGIN with UNGEXP-NATIVE:

  #~(modify-phases %standard-phases
      (add-after 'unpack 'unpack-extra-sources
        (lambda _
          (copy-recursively #+(origin ...)
                            "extra-source-directory"))))

When the build script implementing the phases is... built, this lambda will look
like this:

  (lambda _
    (copy-recursively "/gnu/store/...-extra-sources"
                      "extra-source-directory"))

Roughly, the UNGEXP-NATIVE compiles the origin directory for the host
architecture, then substitutes itself for a string containing the output path of
the compiled store item.

    -- (

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Creating a package using two sources/origins

[dabbede]

Hi (

as you see I'm not yet practical with G-exp and I still have to
understand where exactly they are useful or not.

On Mon, Dec 12, 2022 at 11:43 PM ( <paren@disroot.org> wrote:
>
> Heya,
>
> What you need to do is combine ORIGIN with UNGEXP-NATIVE:
>
>   #~(modify-phases %standard-phases
>       (add-after 'unpack 'unpack-extra-sources
>         (lambda _
>           (copy-recursively #+(origin ...)
>                             "extra-source-directory"))))
>
> When the build script implementing the phases is... built, this lambda will look
> like this:
>
>   (lambda _
>     (copy-recursively "/gnu/store/...-extra-sources"
>                       "extra-source-directory"))
>
> Roughly, the UNGEXP-NATIVE compiles the origin directory for the host
> architecture, then substitutes itself for a string containing the output path of
> the compiled store item.
>
>     -- (

This is a short extract of my custom .scm file:

(define xenomai-version "3.1")
(define xenomai-origin
    (origin
      (method url-fetch)
      (uri (string-append
"https://xenomai.org/downloads/xenomai/stable/xenomai-"
xenomai-version
                          ".tar.bz2"))
      (sha256
        (base32
           "1064l80p9fdbp553mrbin4s7f5qhnwifhfds8a9wl6p6s10alsb4"))))
(define-public my-help
  (package
    (inherit hello)
    (name "my-help")
    (arguments
      #~(modify-phases %standard-phases
      (add-after 'unpack 'unpack-extra-sources
        (lambda _
          (copy-recursively #+xenomai-origin
                            "extra-source-directory")))))))

First of all, I'm not sure whether the gexp that you suggested is
placed correctly within the "arguments" field of the package.
Second, if I try to build the above my-help package I get an exception
with this backtrace:

pcp@PCP3600 ~/guix [env]$ ./pre-inst-env guix build --keep-failed my-help
Backtrace:
In ice-9/boot-9.scm:
  1747:15 19 (with-exception-handler #<procedure 7f52cb297de0 at ic?> ?)
  1752:10 18 (with-exception-handler _ _ #:unwind? _ # _)
In guix/ui.scm:
    449:6 17 (_)
In guix/scripts/build.scm:
    714:5 16 (_)
In srfi/srfi-1.scm:
   673:15 15 (append-map #<procedure 7f52cc86c000 at guix/scripts/b?> ?)
   586:17 14 (map1 ("x86_64-linux"))
In guix/scripts/build.scm:
   716:21 13 (_ _)
In guix/store.scm:
  1382:11 12 (map/accumulate-builds #<store-connection 256.99 7f52d?> ?)
   1300:8 11 (call-with-build-handler #<procedure 7f52cb297d80 at g?> ?)
In guix/scripts/build.scm:
   670:16 10 (_ #<package my-help@2.12.1 /home/pcp/guix/gnu/packages?>)
   659:24  9 (_ #<package my-help@2.12.1 /home/pcp/guix/gnu/package?> ?)
In guix/packages.scm:
  1317:17  8 (supported-package? #<package my-help@2.12.1 /home/pcp?> ?)
In guix/memoization.scm:
    101:0  7 (_ #<hash-table 7f52ccab3e00 0/31> #<package my-help@2?> ?)
In guix/packages.scm:
  1295:37  6 (_)
  1555:16  5 (package->bag _ _ _ #:graft? _)
  1652:22  4 (thunk)
In unknown file:
           3 (_ "my-help-2.12.1" #:system "x86_64-linux" #:source # # ?)
In ice-9/boot-9.scm:
  1685:16  2 (raise-exception _ #:continuable? _)
  1685:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure apply: Apply to non-list: #<gexp (modify-phases
%standard-phases (add-after (quote unpack) (quote
unpack-extra-sources) (lambda _ (copy-recursively #<gexp-input native
#<origin "https://xenomai.org/downloads/xenomai/stable/xenomai-3.1.tar.bz2"
#<content-hash sha256:1064l80p9fdbp553mrbin4s7f5qhnwifhfds8a9wl6p6s10alsb4>
() 7f52ce4af960>:out> "extra-source-directory"))))
/home/pcp/guix/gnu/packages/dabbede.scm:31:6 7f52cc88c9c0>

Finally, if not too much out of scope, can you explain me why the #+
ungexp can accept an origin object instead of a package? I assumed
from the documentation that gexp where used only to reference among
derivation of packages.

Regards

Re: Creating a package using two sources/origins

[(]

[-- Attachment #1: Type: text/plain, Size: 1801 bytes --]

Heya,

On Mon Dec 12, 2022 at 11:58 PM GMT, dabbede@gmail.com wrote:
> (define-public my-help
>   (package
>     (inherit hello)
>     (name "my-help")
>     (arguments
>       #~(modify-phases %standard-phases
>       (add-after 'unpack 'unpack-extra-sources
>         (lambda _
>           (copy-recursively #+xenomai-origin
>                             "extra-source-directory")))))))

Nope, that's not quite correct. Your ARGUMENTS should look like this:

  (arguments
   (list #:phases
         #~(modify-phases %standard-phases
             (add-after 'unpack 'unpack-extra-sources
               (lambda _
                 (copy-recursively #+xenomai-origin
                                   "extra-source-directory"))))))

You can think of ARGUMENTS as a bunch of keyword arguments to pass to the
procedure that runs the build.

> Finally, if not too much out of scope, can you explain me why the #+
> ungexp can accept an origin object instead of a package? I assumed
> from the documentation that gexp where used only to reference among
> derivation of packages.

You've seen a ``.drv'' path being printed out by the CLI, right?  They contain
instructions to build a store item.  Packages are /lowered/ into DERIVATIONs,
but so are ORIGINs, LOCAL-FILEs, COMPUTED-FILEs, and any other kind of
"file-like object" or "lowerable object", such as the object you create with
FILE-APPEND.  The thing they all have it common is that they all have a lowering
procedure defined using DEFINE-GEXP-COMPILER; you can see a few examples of that
in ``guix/gexp.scm'' and ``guix/packages.scm''.  So, you can use UNGEXP and
UNGEXP-NATIVE on anything that has been DEFINE-GEXP-COMPILERed, and its
derivation will be built and its store path substituted.

    -- (

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Creating a package using two sources/origins

[dabbede]

Hi (, thanks a lot.

Nope, that's not quite correct. Your ARGUMENTS should look like this:
>
>   (arguments
>    (list #:phases
>          #~(modify-phases %standard-phases
>              (add-after 'unpack 'unpack-extra-sources
>                (lambda _
>                  (copy-recursively #+xenomai-origin
>                                    "extra-source-directory"))))))
>
> You can think of ARGUMENTS as a bunch of keyword arguments to pass to the
> procedure that runs the build.
>

With this modification it almost works ;-)
The tar.bz2 file is downloaded in the store as file, and then copied in the
build directory with the name  "extra-source-directory" (but it is not a
directory)

You've seen a ``.drv'' path being printed out by the CLI, right?  They
> contain
> instructions to build a store item.  Packages are /lowered/ into
> DERIVATIONs,
> but so are ORIGINs, LOCAL-FILEs, COMPUTED-FILEs, and any other kind of
> "file-like object" or "lowerable object", such as the object you create
> with
> FILE-APPEND.  The thing they all have it common is that they all have a
> lowering
> procedure defined using DEFINE-GEXP-COMPILER; you can see a few examples
> of that
> in ``guix/gexp.scm'' and ``guix/packages.scm''.  So, you can use UNGEXP and
> UNGEXP-NATIVE on anything that has been DEFINE-GEXP-COMPILERed, and its
> derivation will be built and its store path substituted.
>
>     -- (
>

Now it's a little bit more clear, but for sure I have to practice more
before fully understand the type of "things" I'm using in the code.

Regards