LLM - Platform Configuration

This guide provides instructions on how to configure the various LLM providers supported by SearchAI. All configurations are managed in the
<SearchBlox-Installation-path>/webapps/ROOT/WEB-INF/searchai-config.yml file.

Quick Reference

Provider	llm-platform Value	searchblox-llm (Base URL)	openai-api-key	Use Case
OpenAI	openai	Required	Optional	Connecting directly to OpenAI’s official API. Also Recommended for vLLM or any service with an OpenAI-style API
Ollama	ollama	Required	Not used	Connecting to Ollama’s native API
Llama.cpp	llamacpp	Required	Not used	Connecting to a Llama.cpp server instance

OpenAI (Official API)

Use this provider to connect directly to OpenAI’s services. This is the most direct way to access models like GPT-4o, GPT-4, etc.

Configuration:

Set llm-platform to openai.
Update searchblox-llm URL to https://api.openai.com/v1
You must provide your secret key in openai-api-key

Example:

llm-platform: "openai" 
searchblox-llm: "https://api.openai.com/v1"  
openai-api-key: "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 
models: 
chat: "gpt-4o-mini" 
document-enrichment: "gpt-4o" 
# ... other models

OpenAI-Compatible (Recommended for vLLM)

Uses the same OpenAI settings to connect to any service that follows the OpenAI API format, such as vLLM. No actual OpenAI account is needed.

Configuration:

Set llm-platform to openai.
Set searchblox-llm to the base URL of the compatible service.
The openai-api-key is optional and only needed if the third-party service requires one.

Example (for vLLM):

llm-platform: "openai" 
searchblox-llm: "http://localhost:8000/v1" 
openai-api-key: "" 
models: 
chat: "meta-llama/Llama-3-8B-Instruct" 
document-enrichment: "meta-llama/Llama-3-8B-Instruct" 
# Note: Use the model identifier as loaded by your vLLM server

Ollama (Native API)

Use this to connect to Ollama’s original, native API. This is the primary recommended method for using Ollama.

Configuration:

Set llm-platform to ollama.
Set searchblox-llm to the base URL of your Ollama instance.
The openai-api-key is not used.

Example:

llm-platform: "ollama" 
searchblox-llm: "http://localhost:11434" 
openai-api-key: "" 
models: 
chat: "llama3" 
document-enrichment: "mistral" 
# ... other models

Llama.cpp

Use this provider to connect directly to a running llama.cpp server.

Configuration:

Set llm-platform to llamacpp.
Set searchblox-llm to the URL of your llama.cpp server.
The openai-api-key is not used.

Example:

llm-platform: "llamacpp" 
searchblox-llm: "http://localhost:8080" 
openai-api-key: "" 
models: 
chat: "your-local-gguf-model-name" 
document-enrichment: "your-local-gguf-model-name" 
# ... other models