From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id mDrmLRi+FWAGUwAA0tVLHw (envelope-from ) for ; Sat, 30 Jan 2021 20:14:16 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id qKyzKRi+FWBAAgAAbx9fmQ (envelope-from ) for ; Sat, 30 Jan 2021 20:14:16 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1F32A9404E4 for ; Sat, 30 Jan 2021 20:14:16 +0000 (UTC) Received: from localhost ([::1]:35476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5wdJ-0001S4-PZ for larch@yhetil.org; Sat, 30 Jan 2021 15:14:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35034) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5wd9-0001Ru-W8 for guix-patches@gnu.org; Sat, 30 Jan 2021 15:14:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44321) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l5wd8-0003li-BF for guix-patches@gnu.org; Sat, 30 Jan 2021 15:14:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l5wd8-0005eW-5N for guix-patches@gnu.org; Sat, 30 Jan 2021 15:14:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45972] Julia importer? Resent-From: =?UTF-8?Q?Nicol=C3=B2?= Balzarotti Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 30 Jan 2021 20:14:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45972 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: zimoun , Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 45972-submit@debbugs.gnu.org id=B45972.161203760721680 (code B ref 45972); Sat, 30 Jan 2021 20:14:02 +0000 Received: (at 45972) by debbugs.gnu.org; 30 Jan 2021 20:13:27 +0000 Received: from localhost ([127.0.0.1]:55867 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5wcY-0005da-MB for submit@debbugs.gnu.org; Sat, 30 Jan 2021 15:13:27 -0500 Received: from mail-wr1-f51.google.com ([209.85.221.51]:44700) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5wcV-0005dL-3s for 45972@debbugs.gnu.org; Sat, 30 Jan 2021 15:13:25 -0500 Received: by mail-wr1-f51.google.com with SMTP id d16so12320214wro.11 for <45972@debbugs.gnu.org>; Sat, 30 Jan 2021 12:13:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=cw7ZVwwYDHEdKdVBUqkEk5QH+yTYAF7XEdZnwGHjk/M=; b=NcaJ/ewB6hDtZCf5PCGgQtGC/qirdzn1rY2Xs+Uc8BfubWo/fdr1mW0jJJ6D+fgO6N 0E+4dSHZci+iWDdiyAPGkxlV6frGzpoSnsnl77C7u/E5aDr+Cy79gKOkE7Ls9p/VkXIp Cq7fym+P7G/YtHwP/Cpt62A3F2Pu66E9ugQ1G+tqvMHRWPtb+cp1RW1SDPY/NuA7GFWr o+1XnySdKYmdRYN3C8k1fqP7rsTs7IAeYWOfWTI8J2OrtNISPEaB8kuEGWH+t+cNWTSQ IYe29AwVC+9q0L9ipozI/r/VBiq8AqvkvX4PMo9wwuKxPav2zQIVpDkV/6MqxYsPinW9 1E/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=cw7ZVwwYDHEdKdVBUqkEk5QH+yTYAF7XEdZnwGHjk/M=; b=OJyhqRj+Xm0A+U95oTB8DMGrmbYXFj/BivfizA1EouRVxdAYPA5clUeurHiad+5IxD OZfLG38/kvLIfh+ymwUVRvnEnia45aXSIUHhZUA5snNAvqiJJnbPU0Zx3dpMrM7xJKXm G/GioMXXHOb3J3NZ8XO1Oxg9oLM4yOfGGQmKRlCHx1SR6Rf7ur7REM3G9cnpTG/59+mL nOkBqVeno8oxuoJhrnUYXjBJhTKOqrsqixeQmH4sdPeyOgkcZyARO/4xQBj+uzIiSGvR OgmupSmd6/AZ8L5+sVBl/r9/C045exHs7dgDcSXGJwAVcPQtCczLzLepKuLJjiKr6zHi nPZQ== X-Gm-Message-State: AOAM530coatgUa+Wjd2T8DaRZS1LAb59RUrt/TbBMdVmNI4XhHAAanFQ Uv+85pZKSt2pNfdlQsG4xm+uh3Ihc9c= X-Google-Smtp-Source: ABdhPJwQ7M/g1+aUtZ2s5vqqZuBXaqlzAjqySN9/r2fWlferRFNo/1HXQglWghVqb567hxdB0+7u4w== X-Received: by 2002:adf:f183:: with SMTP id h3mr11084909wro.30.1612037597264; Sat, 30 Jan 2021 12:13:17 -0800 (PST) Received: from guixSD (host-87-4-165-57.retail.telecomitalia.it. [87.4.165.57]) by smtp.gmail.com with ESMTPSA id h187sm16023368wmf.30.2021.01.30.12.13.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 Jan 2021 12:13:16 -0800 (PST) From: =?UTF-8?Q?Nicol=C3=B2?= Balzarotti In-Reply-To: <86pn1mfpp1.fsf@gmail.com> References: <87ft2xzu1g.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <87wnvzwfgu.fsf@gnu.org> <87lfcfcmf3.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <87mtwut34z.fsf@gnu.org> <87czxpdhsv.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <86pn1mfpp1.fsf@gmail.com> Date: Sat, 30 Jan 2021 21:13:14 +0100 Message-ID: <8735yi9o9x.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 45972@debbugs.gnu.org Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 0.75 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b="NcaJ/ewB"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 1F32A9404E4 X-Spam-Score: 0.75 X-Migadu-Scanner: scn0.migadu.com X-TUID: JRJUZEzCub53 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable zimoun writes: > Hi, Hi Simon! >> >> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> = println") > > With a bit more glue, could this be transformed into something like > =E2=80=9Cjulia->guix-package=E2=80=9D? And so have a Julia package impor= ter, even if it > fails for some cases. Well, if you mean "Can we use Pkg.jl to generate package definitions for us?" the answer is "probably yes, but I never investigated this". That line uses just Julia Base TOML.jl, which for some reason is defined inside module Pkg.jl (and it's not top-level). If you instead meant "Can we have a Julia importer?" some time ago I wrote one in Julia, I've not used it in the last year, but it did work quite well back then. I attach it here for reference. But before digging into Pkg3, I need to push a new set of packages which contains julia-jllwrappers, which is needed for Julia packages which require binary distributions to work. After that, it will be possible to create an importer that work "ok" even with packages depending on .so libraries. --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=import.jl Content-Description: julia import.jl using Pkg using Pkg.TOML using LibGit2 const base = ["Base64", "CRC32c", "Dates", "DelimitedFiles", "Distributed", "FileWatching", "Future", "InteractiveUtils", "Libdl", "LibGit2", "LinearAlgebra", "Logging", "Markdown", "Mmap", "Pkg", "Printf", "Profile", "Random", "REPL", "Serialization", "SHA", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "SuiteSparse", "Test", "Unicode", "UUIDs"] const disabled = ["WinRPM", "Homebrew", "CMake", "Docile", "Color", "HTTPClient", "ICU", "Calendar", "LegacyStrings", "Nulls"] registrypath = expanduser("~/.julia-old/registries/General/") registry = joinpath(registrypath, "Registry.toml") const register = TOML.parse(join(readlines(registry), "\n")) jlpkgname(info) = "julia-$(lowercase(info.name))" function getrev(path) versions = TOML.parse(join(readlines(path), "\n")) versionv = findmax(map(x -> VersionNumber(x), keys(versions) |> collect)) (rev = versions[string(versionv[1])]["git-tree-sha1"], ver = versionv[1]) end function getpackagebyuuid(uuid) uuid in keys(register["packages"]) || return nothing path = register["packages"][uuid]["path"] getpath(x) = joinpath(registrypath, joinpath(path, x)) package = TOML.parse(join(readlines(getpath("Package.toml")), "\n")) deppath = getpath("Deps.toml") isfile(deppath) || return nothing deps = TOML.parse(join(readlines(deppath), "\n")) (name = package["name"], uuid = package["uuid"], repo = package["repo"], deps = deps, vers = getrev(getpath("Versions.toml"))) end function getpackage(wanted) uuid = findfirst(p -> lowercase(p["name"]) == lowercase(wanted), register["packages"]) uuid == nothing && return nothing return getpackagebyuuid(uuid) end function getdeps!(deps, vers, recursive, out) flat(arr::Array) = mapreduce(x -> isa(x, Array) ? flat(x) : x, append!, arr, init=[]) v = map(p -> VersionNumber.(split(p, '-')), keys(deps) |> collect) valid = findall(x -> length(x) == 1 || (x[2] == v"0" && x[1] <= vers) || x[1] <= vers <= x[2], v) f = flat(map(x -> values(x), values(collect(values(deps))[valid]))) push!.(Ref(out), f) # if recursive # push! # end nothing end function have(info) file = "/home/nixo/git/guix/gnu/packages/julia-xyz.scm" return "(name \"" * jlpkgname(info) * "\")" in strip.(readlines(file)) end function gethash(info) wd = mktempdir() if info.name in base || info.name in disabled || have(info) return "" end println(stderr, "Cloning $(info.name) in $wd") repo = LibGit2.clone(info.repo, wd) hash = cd(wd) do out = Pipe() try LibGit2.checkout!(repo, string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit, LibGit2.GitTag(repo, "v" * string(info.vers.ver)))))) catch e try LibGit2.checkout!(repo, string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit, LibGit2.GitCommit(repo, "v" * string(info.vers.ver)))))) catch e # FIXME: if this happens, return the commit too and use it in the package println(stderr, "Failed to checkout $(e), continuing") end end run(pipeline(`guix hash -rx .`, stdout=out)) readline(out) end rm(wd, recursive = true) hash end function makepackage(info; done = []) if info === nothing @warn "Could not find package (have you cloned the registry?)" return elseif info in done return "" end push!(done, info) deps = String[] getdeps!(info.deps, info.vers.ver, true, deps) # TODO: remove deps that are in base deps = filter(x -> x !== nothing, getpackagebyuuid.(deps)) deplist = join(map(name -> "(\"$name\" ,$name)", jlpkgname.(deps)), '\n') packagedeps = join(makepackage.(deps, done = done), "") hash = gethash(info) hash == "" && return "" """ $packagedeps (define-public $(jlpkgname(info)) (package (name "$(jlpkgname(info))") (version "$(info.vers.ver)") (source (origin (method git-fetch) (uri (git-reference (url "$(info.repo)") (commit (string-append "v" version)))) (file-name "$(info.name)") (sha256 (base32 "$hash")))) (propagated-inputs `($(deplist))) (build-system julia-build-system) (home-page "$(info.repo)") (synopsis "") (description "") (license license:expat))) """ end println.(makepackage.(getpackage.(ARGS))) --=-=-=--