From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: contact@karthinks.com Newsgroups: gmane.emacs.devel Subject: Re: LLM Experiments, Part 1: Corrections Date: Tue, 23 Jan 2024 17:26:48 -0800 Message-ID: <87plxrr06f.fsf@gmail.com> References: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="28720"; mail-complaints-to="usenet@ciao.gmane.io" Cc: sskostyaev@gmail.com To: Andrew Hyatt , emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Wed Jan 24 04:21:52 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rSTpn-0007Ee-Mi for ged-emacs-devel@m.gmane-mx.org; Wed, 24 Jan 2024 04:21:51 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rSTol-0000t1-MS; Tue, 23 Jan 2024 22:20:47 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rSS2l-0004wd-DZ for emacs-devel@gnu.org; Tue, 23 Jan 2024 20:27:07 -0500 Original-Received: from mta-12-4.privateemail.com ([198.54.127.107]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rSS2i-0003VN-4G for emacs-devel@gnu.org; Tue, 23 Jan 2024 20:27:06 -0500 Original-Received: from mta-12.privateemail.com (localhost [127.0.0.1]) by mta-12.privateemail.com (Postfix) with ESMTP id 5DCE618000AE; Tue, 23 Jan 2024 20:26:51 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=karthinks.com; s=default; t=1706059611; bh=1mdGZrVs7SX0JcOHwCEzieteU6XlTtXD5pRi61a03FA=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=oHXqAlOtNQzsZB6eRbYLE5gbMIB/ik/ov/tspUr/HdZSPdh1vnpYdGhJpvjpfCIE+ ndt2q1paFS0qxTldWP0oA39qCe0LSbR40yGSkoxlDOqZ9PKdf3kz3FI9HHUHbngeAr 1aKXRPxkpT6y0VB0tdKxI3003iaqLCdphEuGGal+w8+090rYHJdpYptXIWHU0HXlEi pZu9bZeH6jzY43ekXbxmI4q3Tz36Snjw7XdpMlk+JTvYEo1rPZbDMYTPTIK+lWgt1G IPCOqPdyxOqpkUHviBtI1trwvPJoRAGJvHC5dGosPDudNzYMfcIqLxsnvU9Z1TPE4a c2uVdiVxfkTRw== Original-Received: from localhost (ip68-6-127-223.sb.sd.cox.net [68.6.127.223]) by mta-12.privateemail.com (Postfix) with ESMTPA; Tue, 23 Jan 2024 20:26:49 -0500 (EST) In-Reply-To: X-Virus-Scanned: ClamAV using ClamSMTP Received-SPF: pass client-ip=198.54.127.107; envelope-from=contact@karthinks.com; helo=MTA-12-4.privateemail.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Tue, 23 Jan 2024 22:20:45 -0500 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:315279 Archived-At: Hi Andrew, Having worked on similar problems in gptel for about nine months now (without much success), here are some thoughts. > Question 1: Does the llm-flows.el file really belong in the llm > package? It does help people code against llms, but it expands > the scope of the llm package from being just about connecting to > different LLMs to offering a higher level layer necessary for > these more complicated flows. I think this probably does make > sense, there's no need to have a separate package just for this > one part. I think llm-flows works better as an independent package for now, since it's not clear yet what the "primitive" operations of working with LLMs look like. I suspect these usage patterns will also be a matter of preference for users, as I've realized from comparing how I use gptel to the feature requests I get. > Question 2: What's the best way to write these flows with multiple > stages, in which some stages sometimes need to be repeated? It's > kind of a state machine when you think about it, and there's a > state machine GNU ELPA library already (fsm). I opted to not model > it explicitly as a state machine, optimizing instead to just use > the most straightforward code possible. An FSM might be overkill here. At the same time, I'm not sure that all possible interactions will fit this multi-step paradigm like rewriting text does. > Question 3: How should we deal with context? The code that has the > text corrector doesn't include surrounding context (the text > before and after the text to rewrite), but it usually is helpful. > How much context should we add? The llm package does know about > model token limits, but more tokens add more cost in terms of > actual money (per/token billing for services, or just the CPU > energy costs for local models). Having it be customizable makes > sense to some extent, but users are not expected to have a good > sense of how much context to include. My guess is that we should > just have a small amount of context that won't be a problem for > most models. But there's other questions as well when you think > about context generally: How would context work in different > modes? What about when context may spread in multiple files? It's > a problem that I don't have any good insight into yet. I see different questions here: 1. What should be the default amount of context included with requests? 2. How should this context be determined? (single buffer, across files etc) 3. How should this be different between modes of usage, and how should this be communicated unambiguously? 4. Should we track token costs (when applicable) and communicate them to the user? Some lessons from gptel, which focuses mostly on a chat interface: 1. Users seem to understand gptel's model intuitively since they think of it like a chat application, where the context is expected to be everything in the buffer up to the cursor position. The only addition is to use the region contents instead when the region is is active. This default works well for more than chat, actually. It's good enough when rewriting-in-place or for continuing your prose/code. 2. This is tricky, I don't have any workable ideas yet. In gptel I've experimented with providing the options "full buffer" and "open project buffers" in addition to the default, but these are both overkill, expensive and rarely useful -- they often confuse the LLM more than they help. Additionally, in Org mode documents I've experimented with using sparse trees as the context -- this is inexpensive and can work very well but the document has to be (semantically) structured a certain way. This becomes obvious after a couple of sessions, but the behavior has to be learned nevertheless. 3a. For coding projects I think it might be possible to construct a "sparse tree" with LSP or via treesitter, and send (essentially) an "API reference" along with smaller chunks of code. This should make basic copilot-style usage viable. I don't use LSP or treesitter seriously, so I don't know how to do this. 3b. Communicating this unambiguously to users is a UI design question, and I can imagine many ways to do it. 4. I think optionally showing the cumulative token count for a "session" (however defined) makes sense. > Question 5: Should there be a standard set of user behaviors about > editing the prompt? In another demo (one I'll send as a followup), > with a universal argument, the user can edit the prompt, minus > context and content (in this case the content is the text to > correct). Maybe that should always be the case. However, that > prompt can be long, perhaps a bit long for the minibuffer. Using a > buffer instead seems like it would complicate the flow. Also, if > the context and content is embedded in that prompt, they would > have to be replaced with some placeholder. I think the prompt > should always be editable, we should have some templating system. > Perhaps emacs already has some templating system, and one that can > pass arguments for number of tokens from context would be nice. Another unsolved problem in gptel right now. Here's what it uses currently: - prompt: from the minibuffer - context and content: selected region only The main problem with including context separate from the content here is actually not the UI, it's convincing the LLM to consistently rewrite only the content and use the context as context. Using the prompt+context as the "system message" works, but not all LLM APIs provide a "system message" field. > Question 6: How do we avoid having a ton of very specific > functions for all the various ways that LLMs can be used? Besides > correcting text, I could have had it expand it, summarize it, > translate it, etc. Ellama offers all these things (but without the > diff and other workflow-y aspects). I think these are too much for > the user to remember. Yes, this was the original reason I wrote gptel -- the first few packages for LLM interaction (only GPT-3.5 back then) wanted to prescribe the means of interaction via dedicated commands, which I thought overwhelmed the user while also missing what makes LLMs different from (say) language checkers like proselint and vale, and from code refactoring tools. > It'd be nice to have one function when the > user wants to do something, and we work out what to do in the > workflow. But the user shouldn't be developing the prompt > themselves; at least at this point, it's kind of hard to just > think of everything you need to think of in a good prompt. They > need to be developed, updated, etc. What might be good is a system > in which the user chooses what they want to do to a region as a > secondary input, kind of like another kind of > execute-extended-command. I think having users type out their intention in natural language into a prompt is fine -- the prompt can then be saved and added to a persistent collection. We will never be able to cover (programmatically) even a reasonable fraction of the things the user might want to do. The things the user might need help with is what I'd call "prompt decoration". There are standard things you can specify in a prompt to change the brevity and tone of a response. LLMs tend to generate purple prose, summarize their responses, apologize or warn excessively, etc. We can offer a mechanism to quick-add templates to the prompt to stem these behaviors, or encourage other ones. Karthik