unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Follow-up on julia import script
@ 2023-02-24 16:35 Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-02-24 18:21 ` Simon Tournier
  2023-03-15 12:43 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  0 siblings, 2 replies; 14+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2023-02-24 16:35 UTC (permalink / raw)
  To: guix-devel
  Cc: zimoun, Jean-Baptiste Volatier, Nicolò Balzarotti,
	Ludovic Courtès


Hi all,

I asked a few questions today on the IRC channel about a julia import
script.

Just to summarize and be able to follow the subject later:
- I was looking for a way to write a julia import script base on
information in julia general package repository, which can be quite
simple if we have a functional guile toml/ini parser included (I
couldn't get my parser to work properly, but that might be fixable).

- jvb shared his julia importer code here : https://gitlab.com/vjb/guix.jl

- Apparently Nicolo also has a similer code.

- jvb and zimoun defended that the Julia ecosystem evolves so quickly
  that writing a manual functional julia import script might be
  difficult to maintain, in particular because Julia uses an advanced
  resolver for its compatibilities.

- My approach was a bit different but indeed didn't have the ambition to
  resolve compatibilities between packages (I need to check that but the
  check or the build phase should fail it packages are incompatible,
  thus when trying to build the imported package, we should now which
  packages should be updated if necessary). If we only restrain imports
  to the general package repository, then the task becomes much more
  simple :
  - from the package name, you can generate the uri of the following
  files
  - parse Project.toml to get source information
  - parse Versions.toml to get source hash information
  - parse Deps.toml to get information about dependencies
  Well there is no package description or such, but I think it makes a
  decent start for writing package information. These are all unnested
  toml/ini files, so if we manage to parse them with a simple function,
  I guess we could have such a small / limited importer.
  I do have the very beginning of a script here, but couldn't finish the
  parser :
  https://paste.sr.ht/~ngraves/bbeb74a720e717319344ee3c3e6ea477b378068f/manage

- Maybe we could combine both by
  - using this limited importer if julia is not in the PATH
  - using a proper guix.jl importer if julia and the importer are present in the PATH
  or something like that ?

I do not have a strong opinion about which option is more suitable, but
would greatly profit from having a functional and supported julia import
script. Sorry for inaccuracies if there are some, I do this from memory
mostly.

-- 
Best regards,
Nicolas Graves


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-24 16:35 Follow-up on julia import script Nicolas Graves via Development of GNU Guix and the GNU System distribution.
@ 2023-02-24 18:21 ` Simon Tournier
  2023-02-25 10:11   ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-03-15 12:43 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  1 sibling, 1 reply; 14+ messages in thread
From: Simon Tournier @ 2023-02-24 18:21 UTC (permalink / raw)
  To: Nicolas Graves, guix-devel
  Cc: Jean-Baptiste Volatier, Nicolò Balzarotti,
	Ludovic Courtès

Hi,

Thanks for this fresh input! :-)

On ven., 24 févr. 2023 at 17:35, Nicolas Graves via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> wrote:

> - Apparently Nicolo also has a similer code.

I guess, it is that:

https://yhetil.org/guix/8735yi9o9x.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me


> - My approach was a bit different but indeed didn't have the ambition to
>   resolve compatibilities between packages (I need to check that but the
>   check or the build phase should fail it packages are incompatible,
>   thus when trying to build the imported package, we should now which
>   packages should be updated if necessary). If we only restrain imports
>   to the general package repository, then the task becomes much more
>   simple :
>   - from the package name, you can generate the uri of the following
>   files
>   - parse Project.toml to get source information
>   - parse Versions.toml to get source hash information
>   - parse Deps.toml to get information about dependencies

Based on the General registry, it does not appears to me straightforward
too get all the dependencies; without speaking about some synopsis
and/or description and/or license.

Well, I have tried to implement this: as example consider the Julia
package named Zygote,

https://github.com/JuliaRegistries/General/tree/master/Z/Zygote

Assuming some toml parser available in Guile (it is not too hard using
PEG or a very naive approach, but still :-)), from the file
Package.toml, you get the repository address,

https://github.com/FluxML/Zygote.jl

that you clone.  Then from this, you use the file Project.toml to
consider what is inputs and what is propagated-inputs.  Using README, it
is possible to pre-fill the synopsis and description although it is
never accurate.  You also can try to guess license.  Then, you need to
go back to Registry.  And so on.

Here we are dropping the Compat part and the resolver part.  But as you
said, maybe it would be enough.  Well, we are poorly reimplementing what
is builtin with Julia but why not.

(From my point of view, such effort is not worth.  It appears to me a
better strategy to rely on Julia itself.  Note that “guix import
texlive” requires subversion in the path; so I do not see the point to
reimplement some part of Pkg.jl in Guile instead of reusing it, Julia,
not Guile ;-))

Since this last parenthetical statement is probably not consensual ;-),
we discussed with Nicoló to use Guix extensions and wrap the importer
above relying on Pkg.jl (or the one by Jean-Baptiste; I was not aware :-)).

That’s said, another approach is to parse the JSON:

https://docs.juliahub.com/Zygote/4kbLI/0.6.55/pkg.json

and rely on the service JuliaHub.  This could provide some meta data
such as synopsis, description (based on README) and license.

Well, I stopped in the middle of the first approach but it would not
be that hard to resume. :-)  Or feel free to expand you short first
attempt.

Cheers,
simon


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-24 18:21 ` Simon Tournier
@ 2023-02-25 10:11   ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-02-25 12:19     ` Simon Tournier
  0 siblings, 1 reply; 14+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2023-02-25 10:11 UTC (permalink / raw)
  To: Simon Tournier, guix-devel
  Cc: Jean-Baptiste Volatier, Nicolò Balzarotti,
	Ludovic Courtès


One more question ;)

On 2023-02-24 19:21, Simon Tournier wrote:

>> - Apparently Nicolo also has a similer code.
>
> I guess, it is that:
>
> https://yhetil.org/guix/8735yi9o9x.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me

Thanks!

> Assuming some toml parser available in Guile (it is not too hard using
> PEG or a very naive approach, but still :-)), from the file
> Package.toml, you get the repository address,
>
> https://github.com/FluxML/Zygote.jl
>
> that you clone.  Then from this, you use the file Project.toml to
> consider what is inputs and what is propagated-inputs.

How would you separate inputs from propagated-inputs in this case (and
in Julia in general)?

> That’s said, another approach is to parse the JSON:
>
> https://docs.juliahub.com/Zygote/4kbLI/0.6.55/pkg.json
>
> and rely on the service JuliaHub.  This could provide some meta data
> such as synopsis, description (based on README) and license.

Oh, I didn't know about that, it might indeed be easier, since we
already have the parser, and all information at the same place.

> Cheers,
> simon

--
Best regards,
Nicolas Graves


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-25 10:11   ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
@ 2023-02-25 12:19     ` Simon Tournier
  2023-02-25 18:23       ` Ludovic Courtès
  0 siblings, 1 reply; 14+ messages in thread
From: Simon Tournier @ 2023-02-25 12:19 UTC (permalink / raw)
  To: Nicolas Graves, guix-devel
  Cc: Jean-Baptiste Volatier, Nicolò Balzarotti,
	Ludovic Courtès

Hi Nicolas,

On Sat, 25 Feb 2023 at 11:11, Nicolas Graves via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> wrote:

> How would you separate inputs from propagated-inputs in this case (and
> in Julia in general)?

Well, I am not sure to understand the question.  Basically, Julia
packages require run-time packages and, except very rare cases, we need
to specify them via propagated-inputs.  The the packages required for
running tests are listed with native-inputs.

For instance, consider the package julia-tables, it reads:

    (propagated-inputs
     (list julia-dataapi
           julia-datavalueinterfaces
           julia-iteratorinterfaceextensions
           julia-tabletraits))
    (native-inputs
     (list julia-datavalues
           julia-queryoperators))

when Project.toml reads

--8<---------------cut here---------------start------------->8---
[deps]
DataAPI = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
DataValueInterfaces = "e2d170a0-9d28-54be-80f0-106bbe20a464"
IteratorInterfaceExtensions = "82899510-4779-5014-852e-03e436cf321d"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
OrderedCollections = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
TableTraits = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[...]

[extras]
DataValues = "e7dc6d0d-1eca-5fa6-8ad6-5aecde8b7ea5"
QueryOperators = "2aef5ad7-51ca-5a8f-8e88-e75cf067b44b"
SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
--8<---------------cut here---------------end--------------->8---

Obviously, you need to maintain a hard-coded list of built-in Julia
packages as LinearAlgebra, OrderedCollections or Test.

Does it make sense for your question?

Cheers,
simon


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-25 12:19     ` Simon Tournier
@ 2023-02-25 18:23       ` Ludovic Courtès
  2023-02-27 10:14         ` Simon Tournier
  0 siblings, 1 reply; 14+ messages in thread
From: Ludovic Courtès @ 2023-02-25 18:23 UTC (permalink / raw)
  To: Simon Tournier
  Cc: Nicolas Graves, guix-devel, Jean-Baptiste Volatier,
	Nicolò Balzarotti

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> For instance, consider the package julia-tables, it reads:
>
>     (propagated-inputs
>      (list julia-dataapi
>            julia-datavalueinterfaces
>            julia-iteratorinterfaceextensions
>            julia-tabletraits))
>     (native-inputs
>      (list julia-datavalues
>            julia-queryoperators))
>
> when Project.toml reads
>
> [deps]
> DataAPI = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
> DataValueInterfaces = "e2d170a0-9d28-54be-80f0-106bbe20a464"
> IteratorInterfaceExtensions = "82899510-4779-5014-852e-03e436cf321d"
> LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
> OrderedCollections = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
> TableTraits = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
> Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

How does one resolve those UUIDs?

It seems that one would “just” need a toml parser to put this to work?

The other option you mentioned—using the JuliaHub API—doesn’t sound too
bad to me, and it would be quite easy to implement, similar to (guix
import pypi) and the like I suppose?

Curiously,
Ludo’.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-25 18:23       ` Ludovic Courtès
@ 2023-02-27 10:14         ` Simon Tournier
  2023-02-27 11:37           ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  0 siblings, 1 reply; 14+ messages in thread
From: Simon Tournier @ 2023-02-27 10:14 UTC (permalink / raw)
  To: Ludovic Courtès
  Cc: Nicolas Graves, guix-devel, Jean-Baptiste Volatier,
	Nicolò Balzarotti

Hi,

>> DataAPI = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
>> DataValueInterfaces = "e2d170a0-9d28-54be-80f0-106bbe20a464"
>> IteratorInterfaceExtensions = "82899510-4779-5014-852e-03e436cf321d"
>> LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
>> OrderedCollections = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
>> TableTraits = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
>> Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
>
> How does one resolve those UUIDs?

Arf, good question. :-)

You have to go back to General registry and open the corresponding
entry.  For instance,

    name = "DataAPI"
    uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
    repo = "https://github.com/JuliaData/DataAPI.jl.git"

    https://github.com/JuliaRegistries/General/blob/master/D/DataAPI/Package.toml


What is really boring is the ’compat’ part that I skipped.

Cheers,
simon


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-27 10:14         ` Simon Tournier
@ 2023-02-27 11:37           ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-02-27 13:46             ` Simon Tournier
  2023-03-06 11:23             ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  0 siblings, 2 replies; 14+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2023-02-27 11:37 UTC (permalink / raw)
  To: Simon Tournier, Ludovic Courtès
  Cc: guix-devel, Jean-Baptiste Volatier, Nicolò Balzarotti


What we get from juliahub is really not bad, since it's really close to
what we have in other importers.

I've been working on that this wkend, nearly finished a first version,
but there's still a few points I'm not sure of :

- we don't have a `hash` field in the json, tough we might be able to
  get it by parsing Versions.toml additionally to the json. This I can
  do with a very simple parser.

- there's a difficulty for defining guix-name->julia-name with the
  simple guix/import/utils.scm snake-case. We should define proper
  functions to convert without ambiguity from CamelCase to snake-case
  and back, and that also encompasses cases like JuMP and OCReract. I
  had one option in mind but haven't tested it to get the name in the
  url. Or maybe we should have a #:julia-name field in arguments or
  properties. WDYT ?

- We only have direct and indirect dependencies in juliahub's json. We
  can't separate native-inputs (as is done in jvb1's guix.jl) or
  propagated-inputs (as discussed with zimoun) without additionally
  parsing Project.toml. This I can't do with my very simple parser.

- readme is full of html junk, I don't know if we can process it to get
  pure text output easily in Guile ?

It looks like that as of now :

(define-module (guix scripts juliahub)
  #:use-module (ice-9 textual-ports)
  #:use-module (ice-9 regex)
  #:use-module (ice-9 match)
  #:use-module (ice-9 string-fun)
  #:use-module (srfi srfi-9)
  #:use-module (srfi srfi-1)
  #:use-module (guix http-client)
  #:use-module (guix import utils)
  #:use-module (guix import json)
  #:use-module (guix base16)
  #:use-module (guix base32)
  #:use-module (guix packages)
  #:use-module (json)
  #:use-module ((guix licenses) #:prefix license:)

  #:export (juliahub->guix-package))

(define (juliahub-uri name)
  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
         (port (http-fetch url #:text? #t))
         (_ (get-line port))
         (meta (get-line port))
         (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
         (redirect (match:substring (string-match regex meta))))
    (close-port port)
    (string-drop redirect 4)))

(define (juliahub-url name)
  (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
         (uri (juliahub-uri name)))
    (string-append url uri "/")))

(define (juliahub-slug-version name)
  (let* ((uri (juliahub-uri name))
         (slug (string-take uri 5))
         (latest-version (string-drop uri 6)))
    `(,slug ,latest-version)))

(define (json->juliahub-direct-dependencies vector)
  (if (vector? vector)
      (filter-map
       (lambda (el)
         (let ((dep (json->juliahub-dependency el)))
           (if (juliahub-dependency-direct? dep)
               dep
               #f)))
       (vector->list vector))))

;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
  json->juliahub-package
  (homepage juliahub-package-homepage) ;string
  (readme juliahub-package-readme) ;string
  ;; (slug juliahub-package-slug) ;string
  (version juliahub-package-version) ;string
  (description juliahub-package-description) ;string
  (dependencies
   juliahub-package-dependencies "deps"
   json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
   ;; (lambda (vector)
     ;; (map json->juliahub-dependency (vector->list vector))))
  (url juliahub-package-url) ;string
  (uuid juliahub-package-uuid) ;string
  (license juliahub-package-license)) ;string

(define-json-mapping <juliahub-dependency>
  make-juliahub-dependency juliahub-dependency?
  json->juliahub-dependency
  (direct? juliahub-dependency-direct? "direct") ;boolean
  (name juliahub-dependency-name) ;string
  (uuid juliahub-dependency-uuid) ;string
  (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
  ;; (slug juliahub-dependency-slug) ;string

(define (julia-name->guix-name name)
  (string-append "julia-" (snake-case name)))

(define* (juliahub-fetch name #:key (version #f))
  "Return a <juliahub-package> record for package NAME, or #f on failure."
  (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
         json->juliahub-package))

(define (make-julia-sexp name version uri hash home-page synopsis description
                         dependencies licenses)
  "Return the `package' s-expression for a Julia package with the given NAME,
VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
  `(package
     (name ,(julia-name->guix-name name))
     (version ,version)
     (source (origin
               (method url-fetch)
               (uri ,uri)
               (sha256
                (base32
                 "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
                 ;; ,(bytevector->nix-base32-string hash)
                 ))))
     (build-system julia-build-system)
     ,@(if (null? dependencies)
           '()
           `((inputs
              (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
                           dependencies)))))
     (synopsis ,synopsis)
     (description ,description)
     (home-page ,home-page)
     (license ,(match licenses
                 (() #f)
                 ((license) (license->symbol license))
                 (_ `(list ,@(map license->symbol licenses)))))))

(define* (juliahub->guix-package package-name
                                 #:key version #:allow-other-keys)
  "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
  (let ((package (if version
                      (juliahub-fetch package-name version)
                      (juliahub-fetch package-name))))
    (if package
        (let* ((dependencies-names
                (map juliahub-dependency-name
                     (juliahub-package-dependencies package)))
               (licenses
                (map spdx-string->license
                     (list (juliahub-package-license package)))))
          (values (make-julia-sexp
                   package-name
                   (juliahub-package-version package)
                   (juliahub-package-url package)
                   "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
                   (juliahub-package-homepage package)
                   (juliahub-package-description package)
                   (beautify-description (juliahub-package-readme package))
                   (juliahub-package-dependencies package)
                   licenses)
                  dependencies-names))
        (values #f '()))))


-- 
Best regards,
Nicolas Graves


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-27 11:37           ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
@ 2023-02-27 13:46             ` Simon Tournier
  2023-02-27 18:01               ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-03-06 11:23             ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  1 sibling, 1 reply; 14+ messages in thread
From: Simon Tournier @ 2023-02-27 13:46 UTC (permalink / raw)
  To: Nicolas Graves, Ludovic Courtès
  Cc: guix-devel, Jean-Baptiste Volatier, Nicolò Balzarotti

Hi Nicolas,

On lun., 27 févr. 2023 at 12:37, Nicolas Graves via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> wrote:

> What we get from juliahub is really not bad, since it's really close to
> what we have in other importers.

Oh cool!  Thanks.


> - we don't have a `hash` field in the json, tough we might be able to
>   get it by parsing Versions.toml additionally to the json. This I can
>   do with a very simple parser.

Do you mean Versions.toml from General registry?

Well, I do not think it is straightforward to go from Git SHA1 hash to
NAR SHA256 hash, to put it mildly. :-)  Because two points:

  1. Git uses one way for serializing and NAR is another.
  2. SHA1 is one hash function and SHA256 is another.

1. Git uses one way for serializing and NAR is another.  From
Versions.toml [0], it reads,

--8<---------------cut here---------------start------------->8---
["0.6.41"]
git-tree-sha1 = "3cfdb31b517eec4173584fba2b1aa65daad46e09"
--8<---------------cut here---------------end--------------->8---

0: <https://github.com/JuliaRegistries/General/blob/master/Z/Zygote/Versions.toml>

--8<---------------cut here---------------start------------->8---
$ git -C Zygote.jl log -1 --oneline
4777767 (HEAD, tag: v0.6.41) un-bump version

$ guix hash -S git -H sha1 -f hex -x Zygote.jl
3cfdb31b517eec4173584fba2b1aa65daad46e09
--8<---------------cut here---------------end--------------->8---

Let only replace the serializer from Git to NAR:

--8<---------------cut here---------------start------------->8---
$ guix hash -S nar -H sha1 -f hex -x Zygote.jl
85fc67799543110880acaa8c584d44edb3565b7d
--8<---------------cut here---------------end--------------->8---

2. SHA1 is one hash function and SHA256 is another.  Obviously, it is
impossible to go from one to the other.

--8<---------------cut here---------------start------------->8---
$ guix hash -S nar -H sha256 -f hex -x Zygote.jl
f3d1776c4f3f539987141ad4a57198fd62038e6ef5e9a2eea8ba0819aa916f09
--8<---------------cut here---------------end--------------->8---

However, here the format is ’hex’ and it is possible to convert this
checksum from the format ’hex’ to the format ’nix-base32’.  From this
f3d1776c4f3f539987141ad4a57198fd62038e6ef5e9a2eea8ba0819aa916f09, it is
straightforward to get
02bgj6m1j25sm3pa5sgmds706qpxk1qsbm0s2j3rjlrz9xn7glgk as specified by the
recipe,

--8<---------------cut here---------------start------------->8---
    (version "0.6.41")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/FluxML/Zygote.jl")
             (commit (string-append "v" version))))
       (file-name (git-file-name name version))
       (sha256
        (base32 "02bgj6m1j25sm3pa5sgmds706qpxk1qsbm0s2j3rjlrz9xn7glgk"))))
--8<---------------cut here---------------end--------------->8---


For my point of view, you need to clone the repository to be able to
compute the ’hash’ field.


Cheers,
simon


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-27 13:46             ` Simon Tournier
@ 2023-02-27 18:01               ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-02-28  9:25                 ` Simon Tournier
  0 siblings, 1 reply; 14+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2023-02-27 18:01 UTC (permalink / raw)
  To: Simon Tournier, Ludovic Courtès
  Cc: guix-devel, Jean-Baptiste Volatier, Nicolò Balzarotti

On 2023-02-27 14:46, Simon Tournier wrote:

>
> Oh cool!  Thanks.
>
>> - we don't have a `hash` field in the json, tough we might be able to
>>   get it by parsing Versions.toml additionally to the json. This I can
>>   do with a very simple parser.
>
> Do you mean Versions.toml from General registry?

That's what I meant, but it seems indeed impossible. In the hypothesis
this juliahub script gets merged, what should we do? Really cloning the
repository and getting it or rather just provide a given (false) hash?

-- 
Best regards,
Nicolas Graves


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-27 18:01               ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
@ 2023-02-28  9:25                 ` Simon Tournier
  0 siblings, 0 replies; 14+ messages in thread
From: Simon Tournier @ 2023-02-28  9:25 UTC (permalink / raw)
  To: Nicolas Graves, Ludovic Courtès
  Cc: guix-devel, Jean-Baptiste Volatier, Nicolò Balzarotti

Hi,

On lun., 27 févr. 2023 at 19:01, Nicolas Graves via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> wrote:

> That's what I meant, but it seems indeed impossible. In the hypothesis
> this juliahub script gets merged, what should we do? Really cloning the
> repository and getting it or rather just provide a given (false) hash?

Well, many importers are downloading the source for hashing it, so yes
it appears to me something the importer should do.

Moreover, cloning it would allow to extract various inputs /
propagated-inputs / native-inputs.

Thank you for pushing forward. :-)

Cheers,
simon


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-27 11:37           ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-02-27 13:46             ` Simon Tournier
@ 2023-03-06 11:23             ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-03-07 11:52               ` Simon Tournier
  1 sibling, 1 reply; 14+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2023-03-06 11:23 UTC (permalink / raw)
  To: Simon Tournier, Ludovic Courtès
  Cc: guix-devel, Jean-Baptiste Volatier, Nicolò Balzarotti

On 2023-02-27 12:37, Nicolas Graves wrote:

I have a WIP version which should solve the issues of hash and
native-inputs.

I still have the two following issues, if someone has a hint:

> - there's a difficulty for defining guix-name->julia-name with the
>   simple guix/import/utils.scm snake-case. We should define proper
>   functions to convert without ambiguity from CamelCase to snake-case
>   and back, and that also encompasses cases like JuMP and OCReract. I
>   had one option in mind but haven't tested it to get the name in the
>   url. Or maybe we should have a #:julia-name field in arguments or
>   properties. WDYT ?
>
> - readme is full of html junk, I don't know if we can process it to get
>   pure text output easily in Guile ?

--
Best regards,
Nicolas Graves


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-03-06 11:23             ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
@ 2023-03-07 11:52               ` Simon Tournier
  0 siblings, 0 replies; 14+ messages in thread
From: Simon Tournier @ 2023-03-07 11:52 UTC (permalink / raw)
  To: Nicolas Graves, Ludovic Courtès
  Cc: guix-devel, Jean-Baptiste Volatier, Nicolò Balzarotti

Hi Nicolas,

On Mon, 06 Mar 2023 at 12:23, Nicolas Graves wrote:

> I have a WIP version which should solve the issues of hash and
> native-inputs.

Awesome!


> I still have the two following issues, if someone has a hint:
>
>> - there's a difficulty for defining guix-name->julia-name with the
>>   simple guix/import/utils.scm snake-case. We should define proper
>>   functions to convert without ambiguity from CamelCase to snake-case
>>   and back, and that also encompasses cases like JuMP and OCReract. I
>>   had one option in mind but haven't tested it to get the name in the
>>   url. Or maybe we should have a #:julia-name field in arguments or
>>   properties. WDYT ?

I suggest to track the upstream name with the field ’properties’, e.g.,

    (properties `((upstream-name . "CamelCase")))

as it is done for CRAN/Bioconductor importer; see (guix import cran).


>> - readme is full of html junk, I don't know if we can process it to get
>>   pure text output easily in Guile ?

Yeah, I do not know either.

Cheers,
simon


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-02-24 16:35 Follow-up on julia import script Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-02-24 18:21 ` Simon Tournier
@ 2023-03-15 12:43 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  2023-03-18 11:17   ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  1 sibling, 1 reply; 14+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2023-03-15 12:43 UTC (permalink / raw)
  To: guix-devel
  Cc: zimoun, Jean-Baptiste Volatier, Nicolò Balzarotti,
	Ludovic Courtès, efraim


Hi all!

Took me quite more time than I would've liked, but I have a usable
juliahub scheme import script!

It seems there's still one edge case that isn't covered and revolves
around when Julia packagers don't properly tag their git repos (I've
only seen the case with SnoopPrecompile). There's the possibility to
rely on tree commit hashes from the General repository (since this is a
valid way to identify/store a git repo), but that needs some major
changes in the way latest-repository-commit works. Otherwise, it needs
to be done by hand. It might also not work for subpackages in
directories that are up-to-date on juliahub but not yet on github, I
haven't met this case yet.

I'm sending a patch series in the coming minutes.

-- 
Best regards,
Nicolas Graves


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Follow-up on julia import script
  2023-03-15 12:43 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
@ 2023-03-18 11:17   ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
  0 siblings, 0 replies; 14+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2023-03-18 11:17 UTC (permalink / raw)
  To: guix-devel
  Cc: zimoun, Jean-Baptiste Volatier, Nicolò Balzarotti,
	Ludovic Courtès, efraim


Just a quick reminder ;)

The patch series is 62202. 

I can squash the commits if this is too precise, you can tell me ;)

Have a great week-end. 


On 2023-03-15 13:43, Nicolas Graves wrote:

> Hi all!
>
> Took me quite more time than I would've liked, but I have a usable
> juliahub scheme import script!
>
> It seems there's still one edge case that isn't covered and revolves
> around when Julia packagers don't properly tag their git repos (I've
> only seen the case with SnoopPrecompile). There's the possibility to
> rely on tree commit hashes from the General repository (since this is a
> valid way to identify/store a git repo), but that needs some major
> changes in the way latest-repository-commit works. Otherwise, it needs
> to be done by hand. It might also not work for subpackages in
> directories that are up-to-date on juliahub but not yet on github, I
> haven't met this case yet.
>
> I'm sending a patch series in the coming minutes.

-- 
Best regards,
Nicolas Graves


^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2023-03-18 11:17 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-24 16:35 Follow-up on julia import script Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2023-02-24 18:21 ` Simon Tournier
2023-02-25 10:11   ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2023-02-25 12:19     ` Simon Tournier
2023-02-25 18:23       ` Ludovic Courtès
2023-02-27 10:14         ` Simon Tournier
2023-02-27 11:37           ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2023-02-27 13:46             ` Simon Tournier
2023-02-27 18:01               ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2023-02-28  9:25                 ` Simon Tournier
2023-03-06 11:23             ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2023-03-07 11:52               ` Simon Tournier
2023-03-15 12:43 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2023-03-18 11:17   ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).