LLM Experiments: Part 3: LLMs calling elisp

LLM Experiments: Part 3: LLMs calling elisp

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 6969 bytes --]


Hi all,

I've sent a few messages about experiments with Emacs and LLMs, with the
purpose of getting feedback on design. I've mostly done that, so this
may be the last of my updates on LLM experiments. This one should be
fairly interesting and has some insights that this list may find useful.

To lead with those insights, I'd like to announce that LLMs can now call
elisp functions, and this potentially could increase Emacs usability.
Anywhere where it may be more useful to specify intent than a precise
function argument would be an excellent candidate for this.  Ideas on
where this might be useful would be appreciated.

Another insight is that Elisp function specs and documentation is quite
good, but it doesn't have any fine-grained information. LLMs do a good
job of extracting that information, though. If in the future we wanted
to do things like automate adding more type information, or per-argument
documenation, LLMs should be able to a reasonably good first pass of
that.

So, this all started when I saw that Ollama was going to implement
function calling. This is something that Open AI first supported, then
Gemini supported it (but not as well), and now it's seems like it is
coming to some local models as well. Function calling allows you to
specify functions the LLM knows about, which it then "calls" by sending
back the function name it has decided to execute and the arguments to
that function. This allows you to get structured information about the
LLM in a way that was a bit more painful before.

I implemented this in the llm package, currently just on the
function-calling branch
(https://github.com/ahyatt/llm/tree/function-calling). You can pass in a
list of potential functions, which include the function itself,
documentation, argument list, argument documentation, and whether it is
required. As an example, I've created a function that takes in an elisp
function and inserts the corresponding llm function definition at point.
This is at
https://raw.githubusercontent.com/ahyatt/llm/function-calling/utilities/elisp-to-function-call.el.
But the function definition itself is interesting to see: this is what I
get when I make an llm function definition for switch-to-buffer.

(defconst elisp-to-function-call-switch-to-buffer 
  (make-llm-function-call :function
                          'switch-to-buffer
                          :name
                          "switch_to_buffer"
                          :args
                          (list (make-llm-function-arg :name
                                                       "buffer_or_name"
                                                       :type
                                                       'string
                                                       :description
                                                       "A buffer, a string (buffer name), or nil."
                                                       :required
                                                       t)
                                (make-llm-function-arg :name
                                                       "norecord"
                                                       :type
                                                       'boolean
                                                       :description
                                                       "If non-nil, do not put the buffer at the front of the buffer list and do not make the window displaying it the most recently selected one."
                                                       :required
                                                       t)
                                (make-llm-function-arg :name
                                                       "force_same_window"
                                                       :type
                                                       'boolean
                                                       :description
                                                       "If non-nil, the buffer must be displayed in the selected window when called non-interactively; if that is impossible, signal an error rather than calling ‘pop_to_buffer’."
                                                       :required
                                                       t))
                          :description
                          "Display buffer BUFFER_OR_NAME in the selected
  window. If the selected window cannot display the specified buffer
  because it is a minibuffer window or strongly dedicated to another
  buffer, call ‘pop_to_buffer’ to select the buffer in another window.
  If called interactively, read the buffer name using ‘read_buffer’. The
  variable ‘confirm_nonexistent_file_or_buffer’ determines whether to
  request confirmation before creating a new buffer. See ‘read_buffer’
  for features related to input and completion of buffer names.
  BUFFER_OR_NAME may be a buffer, a string (a buffer name), or nil. If
  BUFFER_OR_NAME is a string that does not identify an existing buffer,
  create a buffer with that name. If BUFFER_OR_NAME is nil, switch to
  the buffer returned by ‘other_buffer’. Return the buffer switched
  to."))

All this was generated by the LLM without any correction by me, and now
we have a much more precise spec for a common elisp function.  However,
it isn't perfect: it incorrectly thinks all arguments are required.  I
could potentially pass in more information such as the actual code for
the function (if there's some easy way to retrieve it), so the LLM can
make better decisions.  Better prompting may also help.

You may notice that I have underscores instead of dashes, which was done
on purpose because function calling is trained on javascript functions,
so making everything look like javascript helps prevent errors.

With this function definition, I can use it to ask the LLM to switch
buffers for me, giving some instruction such as "The main llm project
buffer".  I'm attaching a gif of this to demonstrate. Many functions
such as switch-to-buffer can be passed to the llm (I'm not sure how many
functions would be too much yet), but the LLM can decide when and what
to call as conversation happens.  This specific function is mostly just
to demonstrate what is possible, and is not that useful for most people;
but it could be useful for accessibility.

Besides calling existing elisp function, it's pretty useful for new ones
for specific purposes.  For example, I created one that, given a
vocabulary word, will create an Anki flashcard with a sentence that uses
that word.  These techniques make a lot of things pretty easy, and that
includes some of the things in my last email as well, which now don't
need state machines to do simple structured tasks.

Thanks for reading, I hope this was useful.


[-- Attachment #2: buffer switch demo --]
[-- Type: image/gif, Size: 473751 bytes --]