Add external data to a package

Add external data to a package

[Aristide Doussot]

Hello,

     I am new to Guix and want to add data from an external source to my 
package build. More precisely, I am trying to install the library 
casacore and need to add data in its subdirectory share/casacore/data. I 
can download these data as tar via ftp. For now, my code looks like this :

"
(defineWSRT_Measures
(origin
(method url-fetch)
(uri "[...not sure I can disclose...]/WSRT_Measures.ztar")
(sha256
(base32 "1d9qlcmkpsnw8wm7sidp17iwd2d771nsgjwwp568886g5f239a8k"))
)
)
(define-public casacore
(package
(name "casacore")
(version "3.5.0")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/casacore/casacore.git")
(recursive? #t)
(commit "v3.5.0")))
(file-name (git-file-name name version))
(sha256 (base32 "0zw70slc3681mnadg169avjg7d3jwn1x49xn0022207blcig8g2l"))))
(inputs
(listfftwf fftw openblas boost gcc-toolchain git gsl lapack 
gfortran-toolchain readline flex bison cfitsio wcslib python python-numpy
))
(build-system cmake-build-system)
(arguments (list
#:configure-flags #~(list"-DHELLO_WHO=Guix""-DBUILD_DYSCO=ON")
#:build-type "Release"
#:tests? #f;; some test are failing due to lack of data that are 
downloable via ftp for now testing is disable
#:parallel-build? #t;; TODO : check if it makes a difference in building 
time
#:phases
#~(modify-phases %standard-phases
(add-after 'install'install-data
(lambda* (#:key inputs outputs #:allow-other-keys)
(let*(
(out (assoc-ref outputs "out"))
(data (list-ref(find-files "/gnu/store""WSRT_Measures.ztar")0))
(share-dir (string-append out "/share/casacore/data/"))
)
(mkdir-p share-dir)
(invoke "tar""-xzf"data "-C"share-dir)
#t))))
))
(home-page "http://casacore.github.io/casacore/")
(synopsis "")
(description "A suite of C++ libraries for radio astronomy data 
processing.")
(license license:gpl2+)))

"


And it seems to work but I am sure there are far better ways to do this 
than my "take the first element of an overall find-files on 
/gnu/store/". Can someone teach me these better ways ?

Thanks,
Aristide Doussot

Re: Add external data to a package

[Julian Flake]

Hi,

not really sure if this is, what you asked for, but I solved it by 
defining an additional package (let's call it "extension package" 
or "wsrt-measures") for the extension file(s), similar to what you 
already do. I would unpack the data in the extension package 
already. Then I add that extension package to the inputs of the 
package to extend. In your 'install-file lambda*, you can then 
refer to the extension file(s) via (string-append (assoc-ref 
inputs "wsrt-measures") "/path/to/file/s").

Maybe there are better ways, I'm also interested in.

Best Regards,
nutcase

On Tue, Nov 12 2024, Aristide Doussot wrote:

> Hello,
>
>     I am new to Guix and want to add data from an external 
> source to
> my package build. More precisely, I am trying to install the 
> library
> casacore and need to add data in its subdirectory
> share/casacore/data. I can download these data as tar via 
> ftp. For
> now, my code looks like this :
>
> "
> (defineWSRT_Measures
> (origin
> (method url-fetch)
> (uri "[...not sure I can disclose...]/WSRT_Measures.ztar")
> (sha256
> (base32 "1d9qlcmkpsnw8wm7sidp17iwd2d771nsgjwwp568886g5f239a8k"))
> )
> )
> (define-public casacore
> (package
> (name "casacore")
> (version "3.5.0")
> (source
> (origin
> (method git-fetch)
> (uri (git-reference
> (url "https://github.com/casacore/casacore.git")
> (recursive? #t)
> (commit "v3.5.0")))
> (file-name (git-file-name name version))
> (sha256 (base32 
> "0zw70slc3681mnadg169avjg7d3jwn1x49xn0022207blcig8g2l"))))
> (inputs
> (listfftwf fftw openblas boost gcc-toolchain git gsl lapack
> gfortran-toolchain readline flex bison cfitsio wcslib python
> python-numpy
> ))
> (build-system cmake-build-system)
> (arguments (list
> #:configure-flags #~(list"-DHELLO_WHO=Guix""-DBUILD_DYSCO=ON")
> #:build-type "Release"
> #:tests? #f;; some test are failing due to lack of data that are
>  downloable via ftp for now testing is disable
> #:parallel-build? #t;; TODO : check if it makes a difference in
>  building time
> #:phases
> #~(modify-phases %standard-phases
> (add-after 'install'install-data
> (lambda* (#:key inputs outputs #:allow-other-keys)
> (let*(
> (out (assoc-ref outputs "out"))
> (data (list-ref(find-files "/gnu/store""WSRT_Measures.ztar")0))
> (share-dir (string-append out "/share/casacore/data/"))
> )
> (mkdir-p share-dir)
> (invoke "tar""-xzf"data "-C"share-dir)
> #t))))
> ))
> (home-page "http://casacore.github.io/casacore/")
> (synopsis "")
> (description "A suite of C++ libraries for radio astronomy data
> processing.")
> (license license:gpl2+)))
>
> "
>
>
> And it seems to work but I am sure there are far better ways to 
> do
> this than my "take the first element of an overall find-files on
> /gnu/store/". Can someone teach me these better ways ?
>
> Thanks,
> Aristide Doussot

Re: Add external data to a package

[Parnikkapore via]

A lot of objects can be ungexp-ed into a gexp to substitute in its store path. `<origin>`s are one of them:

```scheme
(define WSRT_Measures
  (origin
   (method url-fetch)
   (uri "[...not sure I can disclose...]/WSRT_Measures.ztar")
   (sha256
     (base32 "1d9qlcmkpsnw8wm7sidp17iwd2d771nsgjwwp568886g5f239a8k"))))

(define-public casacore
  (package
   (name "casacore")
   (version "3.5.0")
   (source
     (origin
       (method git-fetch)
       (uri (git-reference
              (url "https://github.com/casacore/casacore.git")
              (recursive? #t)
              (commit "v3.5.0"))))
    (file-name (git-file-name name version))
    (sha256 (base32 "0zw70slc3681mnadg169avjg7d3jwn1x49xn0022207blcig8g2l")))
   (inputs
     (list fftwf fftw openblas boost gcc-toolchain git gsl lapack gfortran-toolchain readline flex bison cfitsio wcslib python python-numpy))
   (build-system cmake-build-system)
   (arguments (list
               #:configure-flags #~(list "-DHELLO_WHO=Guix" "-DBUILD_DYSCO=ON")
               #:build-type "Release"
               #:tests? #f ;; some test are failing due to lack of data that are downloable via ftp; for now testing is disabled
               #:parallel-build? #t ;; TODO : check if it makes a difference in building time
               #:phases
               #~(modify-phases %standard-phases
                   (add-after 'install 'install-data
                     (lambda* (#:key inputs outputs #:allow-other-keys)
                       (let* ((out (assoc-ref outputs "out"))
                              (data #$WSRT_Measures)  ; <- ===== See here! ================
                              (share-dir (string-append out "/share/casacore/data/")))
                         (mkdir-p share-dir)
                         (invoke "tar" "-xzf" data "-C" share-dir)
                         #t))))))
   (home-page "http://casacore.github.io/casacore/")
   (synopsis "")
   (description "A suite of C++ libraries for radio astronomy data processing.")
   (license license:gpl2+)))
```

Yeah, this is not well-explained in the documentation right now.

(Also took the opportunity to reformat the code from the email. There seems to be quite a few mismatching parentheses, which would've led to some very confusing error messages... my advice is turning on the Lisp / Scheme mode of your editor and turning on the "rainbow parentheses" option.)