From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Andy Tai Newsgroups: gmane.lisp.guile.user Subject: announce: guile_llama_cpp 0.1 release Date: Sun, 2 Jun 2024 21:45:28 -0700 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="8800"; mail-complaints-to="usenet@ciao.gmane.io" To: guile-user@gnu.org Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Mon Jun 03 06:46:48 2024 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sDzap-00028K-4J for guile-user@m.gmane-mx.org; Mon, 03 Jun 2024 06:46:47 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sDzaJ-0004jY-17; Mon, 03 Jun 2024 00:46:15 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sDzaG-0004ij-OH for guile-user@gnu.org; Mon, 03 Jun 2024 00:46:12 -0400 Original-Received: from mail-vk1-xa34.google.com ([2607:f8b0:4864:20::a34]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sDzaB-0001iv-6r for guile-user@gnu.org; Mon, 03 Jun 2024 00:46:12 -0400 Original-Received: by mail-vk1-xa34.google.com with SMTP id 71dfb90a1353d-4eafded3c66so1267992e0c.0 for ; Sun, 02 Jun 2024 21:46:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail-com.20230601.gappssmtp.com; s=20230601; t=1717389965; x=1717994765; darn=gnu.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=YPUUTN4NQGhu6p446uPLAQMzoz6iKws3aHmlCY2QB3o=; b=3eRVOo28TliVQ38W+ZaBjgXCTGShsbascqCDqxhrSGkfbL3qfifcbuV/ZEbO4wiOIg 3+iLnJW/8wLEs68ocq24GZ5QquQMF3zn1TU4L9yrlzHyz99eG3ycj63vmihknHCvRdC4 hKT9g6kcWVIyMUZhuh3p5/U12/Owaro/1jXCJHeESh8F2clGDGeSExzxqtF9kDtiDf1t Cbmdg5Mt88pDIoYDOfOxJh4k/J2Ha1GgOP//K58qAoRM26NtGRrlQQe+0QT7ErVpvOux gIOvybQTNuIIrdrI1wXwTmgLdu6E/X+D3j7Dz2d7D1hRnYOEOmUK6DJvgZlTzo1jGRbT O9QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717389965; x=1717994765; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YPUUTN4NQGhu6p446uPLAQMzoz6iKws3aHmlCY2QB3o=; b=wc610epj9H4eLBLp3k/+lwvWVBUunD2uuxrylJxpMtLVM2sg5dLeXS2lSjjeFl9I8x fNz8NYIhefUW4/ucXMNpGyspjyY5LZ3K/1X/5WnpuazIsRElRaCxQcMpqeTBV9gyk8Nq Nx7jPvMbi0yu/QiKwnmjo96OTWhi1c0tsAY34/xbInYhGIqDcqVoHyXuifXVncIUly5m G3cv6epjSTZepHg/M4I/WrgVU4X6puw8NHfxtgYVTs3ZgH+O+RvmHxdwd3JRmlM+jbfw gbaOfSTBNqgC3L8b6vtlMxki2QTZurIBzeF9g3sdnomJhGzD/9E5gdh09uR5Yyoi+2o2 s4Dg== X-Gm-Message-State: AOJu0YxIgez91Tg5sNruIdE7Gojm5cohjwxtyh+sLEqaLH8dkmsyNG1K JQp7d87YTzw9aAgjHehywtY08hEbY+8z8ThgI/uz+U/Dz4ENLE530o1i/3h4BhIQeQw+qXNLv5E I9jJOv+eDKSoXWuMIkA1kCTWOj5oO85wB X-Google-Smtp-Source: AGHT+IFfQv349Tkgc53dxZE5GaE34FkBI4YzJuPFp3I4bo+Bas6deBnE7vwT8d+x+3Q1rRF8hVx5ziSOmIZMCQF9Ltg= X-Received: by 2002:a05:6102:4412:b0:48b:a80a:1afe with SMTP id ada2fe7eead31-48bc21537bcmr9338362137.14.1717389965125; Sun, 02 Jun 2024 21:46:05 -0700 (PDT) Received-SPF: pass client-ip=2607:f8b0:4864:20::a34; envelope-from=lichengtai@gmail.com; helo=mail-vk1-xa34.google.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.001, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.user:19701 Archived-At: # guile_llama_cpp GNU Guile binding for llama.cpp This is version 0.1, Copyright 2024 Li-Cheng (Andy) Tai, atai@atai.org Available as https://codeberg.org/atai/guile_llama_cpp/releases/download/0.1/guile_llama_cpp-0.1.tar.gz Guile_llama_cpp wraps around llama.cpp APIs so llama.cpp can be accessed from Guile scripts and programs, in a manner similar to llama-cpp-ython allowing the use of llama.cpp in Python programs. Currently a simple Guile script is provided to allow simple "chat" with a LLM in gguf format. ## setup and build guile_llama_cpp is written in GNU Guile and C++ and requires Swig 4.0 or later, GNU guile 3.0, and llama.cpp (obviosuly) installed on your system. >From sources, guile_llama_cpp can be built via the usual GNU convention, export LLAMA_CFLAGS=-I/include export LLAMA_LIBS=-L/lib -lllama ./configure --prefix= make make install Once in the future llama.cpp provides pkg-config support, the first two "export" lines can be omitted. If you are running GNU Guix on your system, you can get a shall with all needed dependencies set up with guix shell -D -f guix.scm and then use the usual configure && make && make install commands to build. ## run To use guile_llama_cpp to chat with a LLM (Large Language Model), you need to first download a LLM in gguf format. See instructions on the web such as https://stackoverflow.com/questions/67595500/how-to-download-a-model-from-huggingface As an example, using a "smaller" LLM "Phi-3-mini" from Microsoft; we would first download the model in gguf format via wget: wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf then you can chat with it, in the build directory: ./pre-inst-env simple.scm "What are the planets?" Phi-3-mini-4k-instruct-q4.gguf The general form to do a chat with a model is to invoke the script scripts/simple.scm simple.scm prompt_text model_file_path in the build directory, pretend the command with ./pre-inst-env as it sets up the needed paths and environment variables for proper guile invocation. Currently, the chat supported is limited; you would see the replies from the LLM cut of after a sentence or so. The outputs length issue will be further addressed in future releases. ## roadmap * support for continuous chat, with long replies * support for expose the LLM as a web end point, using a web server built in Guile, so the LLM can be done via a web interface, to allow chatting with remote users * support for embedding LLMs in Guile programs for scenarios like LLM driven software agents ## license Copyright 2024 Li-Cheng (Andy) Tai atai@atai.org This program is licensed under the GNU Lesser General Public License, version 3 or later, as published by the Free Software Foundation. See the license text in the file COPYING. gde_appmenu is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. Hopefully this program is useful.