From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id UGfHEneW/GOTnQAAbAwnHQ (envelope-from ) for ; Mon, 27 Feb 2023 12:39:35 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id AOC8EneW/GOQGQAAauVa8A (envelope-from ) for ; Mon, 27 Feb 2023 12:39:35 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8F16230CF1 for ; Mon, 27 Feb 2023 12:39:34 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pWbpS-0007hU-SL; Mon, 27 Feb 2023 06:38:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pWbpJ-0006b6-Sl for guix-devel@gnu.org; Mon, 27 Feb 2023 06:37:53 -0500 Received: from 4.mo560.mail-out.ovh.net ([87.98.172.75]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pWbpH-0005sJ-6F for guix-devel@gnu.org; Mon, 27 Feb 2023 06:37:53 -0500 Received: from director1.ghost.mail-out.ovh.net (unknown [10.108.4.200]) by mo560.mail-out.ovh.net (Postfix) with ESMTP id 9C8CE25BEB for ; Mon, 27 Feb 2023 11:37:37 +0000 (UTC) Received: from ghost-submission-6684bf9d7b-q785p (unknown [10.110.208.172]) by director1.ghost.mail-out.ovh.net (Postfix) with ESMTPS id C33861FDF8; Mon, 27 Feb 2023 11:37:35 +0000 (UTC) Received: from ngraves.fr ([37.59.142.108]) by ghost-submission-6684bf9d7b-q785p with ESMTPSA id FCoiJ/+V/GP6/j4ATow75g (envelope-from ); Mon, 27 Feb 2023 11:37:35 +0000 X-OVh-ClientIp: 81.67.140.142 To: Simon Tournier , Ludovic =?utf-8?Q?Court?= =?utf-8?Q?=C3=A8s?= Cc: guix-devel@gnu.org, Jean-Baptiste Volatier , =?utf-8?Q?Nicol=C3=B2?= Balzarotti Subject: Re: Follow-up on julia import script In-Reply-To: <86k003za7z.fsf@gmail.com> References: <87a61358ej.fsf@ngraves.fr> <87o7pilyb4.fsf@gmail.com> <874jra5a2x.fsf@ngraves.fr> <865ybqc4zo.fsf@gmail.com> <874jr9wqml.fsf@gnu.org> <86k003za7z.fsf@gmail.com> Date: Mon, 27 Feb 2023 12:37:33 +0100 Message-ID: <87v8jn49vm.fsf@ngraves.fr> MIME-Version: 1.0 Content-Type: text/plain X-Ovh-Tracer-Id: 10808920583950819993 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: -100 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvhedrudeltddgvdejucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurhephffvvefujghffffkgggtsehttdertddttddtnecuhfhrohhmpefpihgtohhlrghsucfirhgrvhgvshcuoehnghhrrghvvghssehnghhrrghvvghsrdhfrheqnecuggftrfgrthhtvghrnhephedvieefkeelieefjedvudelkeelleelheelleetvefftdduuedtvdetteetieeknecuffhomhgrihhnpehjuhhlihgrhhhusgdrtghomhdpjhhulhhirghhuhgsrdhorhhgnecukfhppeduvdejrddtrddtrddupdefjedrheelrddugedvrddutdeknecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpeeonhhgrhgrvhgvshesnhhgrhgrvhgvshdrfhhrqedpnhgspghrtghpthhtohepuddprhgtphhtthhopehguhhigidquggvvhgvlhesghhnuhdrohhrghdpoffvtefjohhsthepmhhoheeitddpmhhouggvpehsmhhtphhouhht Received-SPF: pass client-ip=87.98.172.75; envelope-from=ngraves@ngraves.fr; helo=4.mo560.mail-out.ovh.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Nicolas Graves From: Nicolas Graves via "Development of GNU Guix and the GNU System distribution." Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1677497975; a=rsa-sha256; cv=none; b=emNRmqCHlwkGD3VwhIHHgZ8Y9m2euc4JIojEUP4aEKNIuK8eYjjGUu346mlZsqFwZ57LkE s3UPtiVFs+KwU1Y7Cb3ES03xywtimf195KKxMpaEyycYYkpaWMPOb9KqPiUQIrECkutIoa A4uIKG0Pst6oJLNkPuYGyXO+4qXdJCU9u+g12VUja4TZEqXBnr4tFKM6+rxYyx5Le8Kr+V OqR0qiP2SGP15Kxrf3VBrVIx9f+p2C+ThqXqRjzqeFX+W/KSAdvc7HS+YE6Q5qNMP4brg/ GjNTb0sBGuRsmBS7k8hNEW3v1Zcddyt9UuQoU6E0ww6X7OiA7AjP56bfJQQ+ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1677497975; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=LmRyFSAxZMxXXyUNMT6BasdxPzm+4JbASPdt7GC3Wsw=; b=Is4OJTkxsrOs3cCH9c0pq8Gf73XK4ahZTuhvMs/WClhXWeKaIV8HezH84bElDsNzKCGgQK 4P+y4sp7fV7qJz1CHeiEjcfjeocCTjq8Gm5llFHJfdQQIb5ZKJIRmCDMZgJjC4no7Eyvl4 fq2r9sCnQg0cN4gQaKF+unGCaexfNGPCxZbExX5z8rGFedga67BaY24vz0ZxMw2iZStpUC uZ+OS+atNPyohAnnsHqc2xtU5cIjgutipGA2F/zNAQq0mQ5mQUmITFKEalJn29P0W6TCmT e+/sutISdED+38IKruWx7iv1gN2XHSOmuLySTIPITRpqI+See4+bDUyqPxfSoA== X-Migadu-Queue-Id: 8F16230CF1 Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -1.22 X-Spam-Score: -1.22 X-TUID: l0YnFi80llxU What we get from juliahub is really not bad, since it's really close to what we have in other importers. I've been working on that this wkend, nearly finished a first version, but there's still a few points I'm not sure of : - we don't have a `hash` field in the json, tough we might be able to get it by parsing Versions.toml additionally to the json. This I can do with a very simple parser. - there's a difficulty for defining guix-name->julia-name with the simple guix/import/utils.scm snake-case. We should define proper functions to convert without ambiguity from CamelCase to snake-case and back, and that also encompasses cases like JuMP and OCReract. I had one option in mind but haven't tested it to get the name in the url. Or maybe we should have a #:julia-name field in arguments or properties. WDYT ? - We only have direct and indirect dependencies in juliahub's json. We can't separate native-inputs (as is done in jvb1's guix.jl) or propagated-inputs (as discussed with zimoun) without additionally parsing Project.toml. This I can't do with my very simple parser. - readme is full of html junk, I don't know if we can process it to get pure text output easily in Guile ? It looks like that as of now : (define-module (guix scripts juliahub) #:use-module (ice-9 textual-ports) #:use-module (ice-9 regex) #:use-module (ice-9 match) #:use-module (ice-9 string-fun) #:use-module (srfi srfi-9) #:use-module (srfi srfi-1) #:use-module (guix http-client) #:use-module (guix import utils) #:use-module (guix import json) #:use-module (guix base16) #:use-module (guix base32) #:use-module (guix packages) #:use-module (json) #:use-module ((guix licenses) #:prefix license:) #:export (juliahub->guix-package)) (define (juliahub-uri name) (let* ((url (string-append "https://docs.juliahub.com/" name "/")) (port (http-fetch url #:text? #t)) (_ (get-line port)) (meta (get-line port)) (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*") (redirect (match:substring (string-match regex meta)))) (close-port port) (string-drop redirect 4))) (define (juliahub-url name) (let* ((url (string-append "https://docs.juliahub.com/" name "/")) (uri (juliahub-uri name))) (string-append url uri "/"))) (define (juliahub-slug-version name) (let* ((uri (juliahub-uri name)) (slug (string-take uri 5)) (latest-version (string-drop uri 6))) `(,slug ,latest-version))) (define (json->juliahub-direct-dependencies vector) (if (vector? vector) (filter-map (lambda (el) (let ((dep (json->juliahub-dependency el))) (if (juliahub-dependency-direct? dep) dep #f))) (vector->list vector)))) ;; Julia package. (define-json-mapping make-juliahub-package juliahub-package? json->juliahub-package (homepage juliahub-package-homepage) ;string (readme juliahub-package-readme) ;string ;; (slug juliahub-package-slug) ;string (version juliahub-package-version) ;string (description juliahub-package-description) ;string (dependencies juliahub-package-dependencies "deps" json->juliahub-direct-dependencies) ;list of ;; (lambda (vector) ;; (map json->juliahub-dependency (vector->list vector)))) (url juliahub-package-url) ;string (uuid juliahub-package-uuid) ;string (license juliahub-package-license)) ;string (define-json-mapping make-juliahub-dependency juliahub-dependency? json->juliahub-dependency (direct? juliahub-dependency-direct? "direct") ;boolean (name juliahub-dependency-name) ;string (uuid juliahub-dependency-uuid) ;string (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings ;; (slug juliahub-dependency-slug) ;string (define (julia-name->guix-name name) (string-append "julia-" (snake-case name))) (define* (juliahub-fetch name #:key (version #f)) "Return a record for package NAME, or #f on failure." (and=> (json-fetch (string-append (juliahub-url name) "pkg.json")) json->juliahub-package)) (define (make-julia-sexp name version uri hash home-page synopsis description dependencies licenses) "Return the `package' s-expression for a Julia package with the given NAME, VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES." `(package (name ,(julia-name->guix-name name)) (version ,version) (source (origin (method url-fetch) (uri ,uri) (sha256 (base32 "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5" ;; ,(bytevector->nix-base32-string hash) )))) (build-system julia-build-system) ,@(if (null? dependencies) '() `((inputs (list ,@(map (compose julia-name->guix-name juliahub-dependency-name) dependencies))))) (synopsis ,synopsis) (description ,description) (home-page ,home-page) (license ,(match licenses (() #f) ((license) (license->symbol license)) (_ `(list ,@(map license->symbol licenses))))))) (define* (juliahub->guix-package package-name #:key version #:allow-other-keys) "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the `package' s-expression corresponding to that package, or #f on failure. Optionally include a VERSION string to fetch a specific version juliahub." (let ((package (if version (juliahub-fetch package-name version) (juliahub-fetch package-name)))) (if package (let* ((dependencies-names (map juliahub-dependency-name (juliahub-package-dependencies package))) (licenses (map spdx-string->license (list (juliahub-package-license package))))) (values (make-julia-sexp package-name (juliahub-package-version package) (juliahub-package-url package) "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5" (juliahub-package-homepage package) (juliahub-package-description package) (beautify-description (juliahub-package-readme package)) (juliahub-package-dependencies package) licenses) dependencies-names)) (values #f '())))) -- Best regards, Nicolas Graves