Draft: New models interface

[Mini256]

TODO

• Using LiteLLM as the universal interface to call LLM / Embedding Model / Reranker Model
• Provide more flexible model configuration methods
  • Configure the model provider (credentials) instead of the model in the settings.
  • Resolve the model using provider_name + model_name.
  • Allow override the model configurations on the calling side, such as: temperature, max_token.

Current State

• When supporting a new model provider, we not only need to add the LlamaIndex dependency but also check its compatibility with DSPy.
• Since Autoflow didn't maintain a list of supported models, the users need to enter the exact model name in the input field rather than selecting from a list.
• Even if adding two models from the same provider, the users still have to go through the credentials configuration process again.

Why LiteLLM

• LiteLLM has an active open-source community, making it more timely and easier to adapt to new models.
• LiteLLM supports a comprehensive range of API endpoints, including Text Completions, Chat Completions, Embeddings, and the Rerank endpoint.
• DSPy has already integrated LiteLLM as a unified interface layer for connecting to LLMs.

Sample Code

Using LLM

llm = LLM(
  model="openai/gpt4o",
  api_key=env.OPENAI_API_KEY
)
llm.completion(
  messages=[
    {
       "role": "system",
       "content": "Your are a TiDB expert"
    },
    {
       "role": "user",
       "content": "Does TiDB support Vector Search?"
    }
  ]
)

Get LLM from config

llm_config = kb.config.llm
llm = resolve_llm(
  provider=llm_config.provider
  model=llm_config.model
)

Convert to DSPy LM for use

lm = llm.as_dspy_lm()
qa = dspy.ChainOfThought('question -> answer')
response = qa(question="What is TiDB?")
print(response.answer)

[sykp241095]

How about opening a new branch for refactor this?