all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* LLM Experiments, Part 1: Corrections
@ 2024-01-22 12:57 Psionic K
  2024-01-22 20:21 ` Andrew Hyatt
  0 siblings, 1 reply; 22+ messages in thread
From: Psionic K @ 2024-01-22 12:57 UTC (permalink / raw)
  To: ahyatt, Emacs developers

> I think things have to be synchronous here.

 Snapshot isolation is the best strategy for merging here.  We don't
know what user commands affected the region in question, so using undo
states to merge might need to undo really arbitrary user commands.  To
snapshot isolate, basically you store a copy of the buffer text and
hold two markers where that text was.  You can merge the result if it
arrives on time and then diff the snapshot with the buffer text
between the markers.  If things are too different for a valid merge,
you can give up and drop the results.

These days various CRDT (conflict-free replicated data type)
treatments have great insights into dealing with much worse problems
of multiple asynchronous writers, and it's a good place to look.
There is a crdt.el package for some inspiration.

But definitely not synchronous.

As a package author, I would want to treat my LLM like a fancy
process.  I create it, I handle results.  I have a merging strategy
(this is mainly up to the client, not the library), but I don't care
about the asynchronous details and I don't want to be tied to each
call.

> Question 6

A rock solid library that sticks to the domain model is best for
ecosystem growth.  When that doesn't happen, we get four or five 75%
finished packages because every author is having to figure out and
integrate their high level features with so many backends.  If you
want to work on high level things, build a client for your library and
experience both sides.

Every model will have some mostly static configuration, dynamic
arguments that could change all the time but in practice change just a
few times, and then the input data.  The static configuration, if
absolutely necessary, can be updated for one call via dynamic binding.
The dynamic arguments should be abstracted into a "context" object
that the model backend figures out how to translate into a valid API
call.  The input data is an arbitrary bundle of whatever that model
type consumes as input.  The library user will want to get a valid
context of the dynamic arguments from the library, enabling them to
make changes to it in subsequent calls, but they don't really want to
touch it that much.

As a package author, I would want to focus on integrating outputs and
piping in inputs.  I don't want to write a UI for tuning the model
parameters.  If the model can ask the user to make adjustments and
just give me a record of their decision I can use later, that would be
fantastic.  I should be able to integrate more closely with backends I
know about but otherwise just call with the provided context and my
inputs.

Providers offer multiple models.  As a library user, it's inconvenient
if I have to go through long incantations to get each context that
represents the capability to make valid calls for the provider.  I
want to initialize once and then use an existing context to pull out
the correct context based on the input or output type I need, and then
make refinements that are specific to a call, such as changing quality
or entropy etc.  Input or output type and settings that tune the call
are two different things.  Settings are mostly provider-specific
argument data that doesn't affect the validity of connecting one model
to another.  Input and output type affect which pipes can be connected
to which other pipes.  This distinction between input or output types
and other arguments become important in composition.  I should be able
to connect any string to string model with any other model that
handles strings no matter what the other settings are.

Integrating these systems will be more like distributed streaming
programming than feeding inputs to a GPU with tight synchronization
and everything under our watch, although a local model might work that
way inside its own box.  We should treat them like unreliable external
services.  A call to the model is a command.  When I send a command, I
should store how to handle the reply, but I shouldn't couple myself to
it with nested callbacks or async, which we fortunately don't have
anyway.  The call data just goes into a pile.  If the reply shows up
and it matches a call, we handle it.  If things time out, we
dead-letter and drop the record of making a call.  This is a very good
way to get around the limitations of the process as our main
asynchronous primitive for now.  It works for big distributed services
which by their very nature cannot lock each other or share memory.  It
will work for connecting many models to each other.



^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2024-05-20 17:28 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <m2il3mj961.fsf@gmail.com>
2024-01-22 18:50 ` LLM Experiments, Part 1: Corrections Sergey Kostyaev
2024-01-22 20:31   ` Andrew Hyatt
2024-01-22 22:06     ` T.V Raman
2024-01-23  0:52       ` Andrew Hyatt
2024-01-23  1:57         ` T.V Raman
2024-01-23  3:00         ` Emanuel Berg
2024-01-23  3:49           ` Andrew Hyatt
2024-01-23  1:36 ` João Távora
2024-01-23  4:17   ` T.V Raman
2024-01-23 19:19   ` Andrew Hyatt
2024-01-24  1:26 ` contact
2024-01-24  4:17   ` T.V Raman
2024-01-24 15:00     ` Andrew Hyatt
2024-01-24 15:14       ` T.V Raman
2024-01-24 14:55   ` Andrew Hyatt
2024-01-24  2:28 ` Karthik Chikmagalur
2024-05-20 17:28 ` Juri Linkov
2024-01-22 12:57 Psionic K
2024-01-22 20:21 ` Andrew Hyatt
2024-01-23  6:49   ` Psionic K
2024-01-23 15:19     ` T.V Raman
2024-01-23 19:36     ` Andrew Hyatt

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.