From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id SLW+FpTJ3mYgYwEAe85BDQ:P1 (envelope-from ) for ; Mon, 09 Sep 2024 10:10:28 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id SLW+FpTJ3mYgYwEAe85BDQ (envelope-from ) for ; Mon, 09 Sep 2024 12:10:28 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=p9mgsyNC; dkim=fail ("headers rsa verify failed") header.d=rimm.ee header.s=herman header.b="PWocxdl/"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1725876628; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=pAXYWy5sY5MFa0A/nAkpRCbArKn/rAh3TNrexIQyqH4=; b=PJ8KeHxq4xt2ytaQZ3oI9RFAXO0cbT5Ln/xLpfe2N274k5QVgdGDq8PzLz+omXTKwsRbPP 1uIoNcht//OCThSKoHuP07idMCAdf9fOTlkLW5MWlotXJlljEStpXDuO4YVWs3ujOLY2/S 5pg/d+aozPpFwMDt+5zYD1WLupEApeCALiqS5by/HEj5Kj6/7VGCWO1JXnMVypcwvFc7E1 geP91AgNwdstXbsJC1VqIXVNXr/goDGeX6nmWwbG3DIdBT7SGl/H8L812+lkUc80+/K5rR G8mj0SrELkb3kpJsJDt5ew8znTtAbyhVJbTHLokT9wWK4+gPA5h1rMVuOqKHcQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=p9mgsyNC; dkim=fail ("headers rsa verify failed") header.d=rimm.ee header.s=herman header.b="PWocxdl/"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1725876628; a=rsa-sha256; cv=none; b=Zpn2my53f6PStBJhWEoYl0nJtmVMz7J9Sn58d37o2BQHrI88Thz1cy3ZBiGtY7RdDUUg5M lynXVkzVe8DWk/Duk2f4tx1Tg2EZqjLDZAgAEhJjMlT4xRiBAsCVBiqsZQPE1MWxRtdZYg xGddshOfmF8FkXDTPtVjhLTlzRy+TCrC8LKxQXIcEQedp2hed69Y9g2VLo4yyYSxhucvah m20yz6pPw/jyDjODdF4Jde6mumVpSGfuwCcSB3wUY/yTZSzebiZwlLVmZruHPAqtVTVvnR sr4RB/7xIvpLf8HG4MXGA44OGMdy3udJkNvc8PtFgRDpEP9+eBufbJsImqoK6w== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0ABF2E4FD for ; Mon, 09 Sep 2024 12:10:28 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snbLa-0004yq-To; Mon, 09 Sep 2024 06:10:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snbLP-0004qn-6x for guix-patches@gnu.org; Mon, 09 Sep 2024 06:10:03 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snbLO-0000DY-Qe for guix-patches@gnu.org; Mon, 09 Sep 2024 06:10:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=pAXYWy5sY5MFa0A/nAkpRCbArKn/rAh3TNrexIQyqH4=; b=p9mgsyNCMu2mImGAcranQuGh/N+DMmOeYAW2sPX5O0+mVv1bUwlZER3MRCUnkqJGwaQCY3GWhyJbpbAsJNumwzOpuRNxOoagQNxkAGfwQ0hUgMwr+wV56BMg56sfnKLldgX+DFZKQ60mudGvvxg9xUOahQ911BKpB3cvP2D5jhX4ARSwmAI7lcvi9foWPWmbYQBwWAFfpTqBf2kiVQy30PCB/Bxatk4StWRAhCkfwLZhm1kI3JalTd9Ln1SaOGlvhGWWjHNR1NXoXzr1QwNzGlH+b0CqArAB+KzqQzfFl/lYhC1aUC3RXWljI9xL1sGkzBj4mhIJvbIUQr+XN8D+9w==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1snbLS-0003Sl-5Y for guix-patches@gnu.org; Mon, 09 Sep 2024 06:10:06 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#73094] [PATCH v3 10/10] gnu: Add python-safetensors. Resent-From: Herman Rimm Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 09 Sep 2024 10:10:06 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73094 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 73094@debbugs.gnu.org Cc: Nicolas Graves Received: via spool by 73094-submit@debbugs.gnu.org id=B73094.172587655813157 (code B ref 73094); Mon, 09 Sep 2024 10:10:06 +0000 Received: (at 73094) by debbugs.gnu.org; 9 Sep 2024 10:09:18 +0000 Received: from localhost ([127.0.0.1]:60699 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1snbKf-0003Q2-E5 for submit@debbugs.gnu.org; Mon, 09 Sep 2024 06:09:18 -0400 Received: from 81-205-150-117.fixed.kpn.net ([81.205.150.117]:45303 helo=email.rimm.ee) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1snbKa-0003Nt-J0 for 73094@debbugs.gnu.org; Mon, 09 Sep 2024 06:09:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rimm.ee; s=herman; t=1725876538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pAXYWy5sY5MFa0A/nAkpRCbArKn/rAh3TNrexIQyqH4=; b=PWocxdl/CYPBoIunxLTM6ECM9yQBchWAD6Xmq2tPcFqawRQGNugJ7Q1KIxbu4pppS4g3ao 7P1Zh+Iu7pF86dBgwuWxciLdvfAnvwxK6XVMvQazZfCW85kPOAAhN6MsAyNMGaOK5yGQQU S/EgQ13EOdUI/VwQgJAr+iwmopZSW9AzVtlu8Yu+P5CoZ30YWF8S08KU64T6iEkKcdfMhe efLF76n+oDgQW9wMG6bcxh/7xN3ghIhy5wHBUeDA1VCX/rSQJsL4DzGic/bqHcHOSO/1Vt N/6DNAf9Ama3LkLXkVRThRocldAw+dy3s50S5h3Cw6n6ZFtBTZn5jzog58gl/A== Received: by 81-205-150-117.fixed.kpn.net (OpenSMTPD) with ESMTPSA id b7e45206 (TLSv1.3:TLS_CHACHA20_POLY1305_SHA256:256:NO); Mon, 9 Sep 2024 10:08:58 +0000 (UTC) Date: Mon, 9 Sep 2024 12:08:03 +0200 Message-ID: <20240909100804.13552-11-herman@rimm.ee> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240909100804.13552-1-herman@rimm.ee> References: <20240909100804.13552-1-herman@rimm.ee> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Herman Rimm X-ACL-Warn: , Herman Rimm via Guix-patches From: Herman Rimm via Guix-patches via Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Spam-Score: -6.49 X-Migadu-Queue-Id: 0ABF2E4FD X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -6.49 X-TUID: n/Zrmj3ZUtm/ From: Nicolas Graves * gnu/packages/machine-learning.scm (python-safetensors): New variable. Change-Id: I90a1684d06756ce87ca0862d745a75be5919f0b2 --- gnu/packages/machine-learning.scm | 100 ++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm index 410b71b061..8b9dd2f7e4 100644 --- a/gnu/packages/machine-learning.scm +++ b/gnu/packages/machine-learning.scm @@ -1085,6 +1085,106 @@ (define-public rust-safetensors @code{PyTorch} counterparts.") (license license:asl2.0))) +(define-public python-safetensors + (package + (name "python-safetensors") + (version "0.4.3") + (source + (origin + (method url-fetch) + (uri (pypi-uri "safetensors" version)) + (sha256 + (base32 "1hhiwy67jarm70l0k26fs1cjhzkgzrh79q14bklj2yp0qi8gr19g")) + (modules '((guix build utils) + (ice-9 ftw))) + (snippet + #~(begin ;; Only keeping bindings. + (for-each + (lambda (file) + (unless (member file '("." ".." "bindings" "PKG-INFO")) + (delete-file-recursively file))) + (scandir ".")) + (for-each + (lambda (file) + (unless (member file '("." "..")) + (rename-file (string-append "bindings/python/" file) + file))) + (scandir "bindings/python")))))) + (build-system cargo-build-system) + (arguments + (list + #:modules '((guix build cargo-build-system) + (guix build utils) + (ice-9 regex) + (ice-9 textual-ports) + (srfi srfi-26)) + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack-rust-crates 'inject-safetensors + (lambda _ + (substitute* "Cargo.toml" + (("\\[dependencies\\]") + (format #f "[dependencies]~%safetensors = ~s" + #$(package-version rust-safetensors)))) + (call-with-input-file "Cargo.toml" + (lambda (port) + (let* ((content (get-string-all port)) + (top-match (string-match + "\\[dependencies.safetensors" + content))) + (call-with-output-file "Cargo.toml" + (cut display (match:prefix top-match) <>))))))) + (add-before 'check 'install-rust-library + (lambda _ + (copy-file "target/release/libsafetensors_rust.so" + "py_src/safetensors/_safetensors_rust.so"))) + (replace 'check + (lambda _ + (invoke "python3" + "-c" (string-append "import sys; sys.path.append" + "(\"" (getcwd) "/py_src\")") + "-m" "pytest" + "-n" "auto" + "--dist=loadfile" + "-s" "-v" "./tests/" + ;; Missing jax dependency + "--ignore=./tests/test_flax_comparison.py"))) + (add-after 'install 'install-python + (lambda _ + (let* ((pversion #$(version-major+minor + (package-version python))) + (lib (string-append #$output "/lib/python" pversion + "/site-packages/")) + (info (string-append lib "safetensors-" + #$(package-version this-package) + ".dist-info"))) + (mkdir-p info) + (copy-file "PKG-INFO" (string-append info "/METADATA")) + (copy-recursively + "py_src/safetensors" + (string-append lib "safetensors")))))) + #:cargo-inputs + `(("rust-pyo3" ,rust-pyo3-0.21) + ("rust-memmap2" ,rust-memmap2-0.9) + ("rust-safetensors" ,rust-safetensors) + ("rust-serde-json" ,rust-serde-json-1)))) + (inputs + (list rust-safetensors)) + (native-inputs + (list python-h5py + python-minimal + python-numpy + python-pytest + python-pytest-xdist + python-pytorch + tensorflow)) + (home-page "https://huggingface.co/docs/safetensors") + (synopsis "Simple and safe way to store and distribute tensors") + (description "This package provides a fast (zero-copy) and safe +(dedicated) format for storing tensors safely. This package builds upon +@code{rust-safetensors} and provides Python bindings.") + (license license:asl2.0))) + (define-public python-sentencepiece (package (name "python-sentencepiece") -- 2.45.2