From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>
To: 62443@debbugs.gnu.org
Cc: ngraves@ngraves.fr
Subject: [bug#62443] [PATCH 1/3] gnu: Add sentencepiece.
Date: Sat, 25 Mar 2023 16:32:18 +0100 [thread overview]
Message-ID: <20230325153220.26027-1-ngraves@ngraves.fr> (raw)
In-Reply-To: <875yaoc1nj.fsf@ngraves.fr>
* gnu/packages/machine-learning.scm (sentencepiece): New variable.
---
gnu/packages/machine-learning.scm | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 37d4ef78ad..f6996af77b 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -583,6 +583,33 @@ (define openfst-for-vosk
'("--enable-shared" "--enable-far" "--enable-ngram-fsts"
"--enable-lookahead-fsts" "--with-pic" "--disable-bin")))))
+(define-public sentencepiece
+ (package
+ (name "sentencepiece")
+ (version "0.1.97")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/google/sentencepiece")
+ (commit (string-append "v" version))))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32 "1kzfkp2pk0vabyw3wmkh16h11chzq63mzc20ddhsag5fp6s91ajg"))))
+ (build-system cmake-build-system)
+ (arguments '(#:tests? #f))
+ (native-inputs (list gperftools))
+ (home-page "https://github.com/google/sentencepiece")
+ (synopsis "Unsupervised tokenizer for Neural Network-based text generation")
+ (description "SentencePiece is an unsupervised text tokenizer and
+detokenizer mainly for Neural Network-based text generation systems where the
+vocabulary size is predetermined prior to the neural model training.
+SentencePiece implements subword units (e.g., byte-pair-encoding
+(BPE) and unigram language model) with the extension of direct training from
+raw sentences. SentencePiece allows us to make a purely end-to-end system
+that does not depend on language-specific pre/postprocessing.")
+ (license license:asl2.0)))
+
(define-public shogun
(package
(name "shogun")
--
2.39.2
next prev parent reply other threads:[~2023-03-25 15:33 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-25 15:05 [bug#62443] LLaMA.cpp Nicolas Graves via Guix-patches via
2023-03-25 15:32 ` Nicolas Graves via Guix-patches via [this message]
2023-03-25 15:32 ` [bug#62443] [PATCH 2/3] gnu: Add python-sentencepiece Nicolas Graves via Guix-patches via
2023-03-25 15:32 ` [bug#62443] [PATCH 3/3] gnu: Add llama-cpp Nicolas Graves via Guix-patches via
2023-04-08 12:07 ` [bug#62443] LLaMA.cpp Nicolas Goaziou
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230325153220.26027-1-ngraves@ngraves.fr \
--to=guix-patches@gnu.org \
--cc=62443@debbugs.gnu.org \
--cc=ngraves@ngraves.fr \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).