Re: LLM Experiments, Part 1: Corrections

[Andrew Hyatt <ahyatt@gmail.com>]

 
On 22 January 2024 14:06, "T.V Raman" <raman@google.com> wrote: 

    Some more related thoughts below, mostly thinking aloud: 1. 
    From using gptel and ellama against the same model, I see 
    different 
       style responses, and that kind of inconsistency would be 
       good to get a handle on; LLMs are difficult enough to 
       figure out re what they're doing without this additional 
       variation.

Is this keeping the prompt and temperature constant?  There's 
inconsistency, though, even keeping everything constant due to the 
randomness of the LLM.  I often get very different results, for 
example, to make the demo I shared, I had to run it like 5 times 
because it would either do things too well (no need to demo 
corrections), or not well enough (for example, it wouldn't follow 
my orders to put everything in one paragraph).
 
    2. Package LLM has the laudible goal of bridgeing between 
    models and 
       front-ends, and this is going to be vital. 
    3. (1,2) above lead  to the following question: 4. Can we 
    write down  a list of common configuration vars --- here 
      common across the model axis. Make  it a union of all such 
      params.

I think the list of common model-and-prompt configuration should 
already be already in the llm package already, but we probably 
will need to keep expanding this.

 
    5. Next, write down a list of all configurable params on the 
    UI side.

This will change quite a bit depending on the task. It's unclear 
how much should be configurable - for example, in the demo, I have 
ediff so the user can see and evaluate the diff. But maybe that 
should be configurable, so if the user wants to see just a diff 
output instead, perhaps that should be allowed? When I was 
thinking about a state machine, I was thinking that parts of the 
state machine might be overridable by the user, such as a "have 
the user check the results of the operation" is a state in the 
state machine that the user can just define their own function 
for.  I suspect we'll have a better idea of this after a few more 
demos.
 
    6. When stable, define a single data-structure in elisp that 
    acts as 
       the bridge between the front-end emacs UI and the LLM 
       module.

If I understand you correctly, this would be the configuration you 
listed in your point (4) and (5)?

 
    7. Finally factor out  the settings of that structure and make 
    it 
       possible to create "profiles" so that one can predictably 
       experiment across front-ends and models. 

I like this idea, thanks!