LLM - Platform Configuration

This guide provides instructions on how to configure the various LLM providers supported
by SearchAI. All configurations are managed in the
<SearchBlox-Installation-path>/webapps/ROOT/WEB-INF/searchai-config.yml file.

Quick Reference

Provider	llm-platform Value	searchblox-llm (Base URL)	openai-api-key	Use Case
OpenAI	openai	Required	Optional	Connecting directly to OpenAI’s official API. Also Recommended for vLLM or any service with an OpenAI-style API
Ollama	ollama	Required	Not used	Connecting to Ollama’s native API
Llama.cpp	llamacpp	Required	Not used	Connecting to a Llama.cpp server instance

OpenAI (Official API)

Use this provider to connect directly to OpenAI’s services. This is the most direct way to
access models like GPT-4o, GPT-4, etc.

Configuration:

Set llm-platform to openai.
Update searchblox-llm URL to https://api.openai.com/v1
You must provide your secret key in openai-api-key

Example:

llm-platform: "openai" 
searchblox-llm: "https://api.openai.com/v1"  
openai-api-key: "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 
models: 
chat: "gpt-4o-mini" 
document-enrichment: "gpt-4o" 
# ... other models

OpenAI-Compatible (Recommended for vLLM)

Uses the same OpenAI provider to connect to any service offering an OpenAI-compatible API
endpoint. It’s the recommended method for using local models with vLLM and other similar
model servers.

Configuration:

Set llm-platform to openai.
Set searchblox-llm to the base URL of the compatible service.
The openai-api-key is optional and only needed if the third-party service requires one.

Example (for vLLM):

llm-platform: "openai" 
searchblox-llm: "http://localhost:8000/v1" 
openai-api-key: "" 
models: 
chat: "meta-llama/Llama-3-8B-Instruct" 
document-enrichment: "meta-llama/Llama-3-8B-Instruct" 
# Note: Use the model identifier as loaded by your vLLM server

Ollama (Native API)

Use this to connect to Ollama’s original, native API. This is the primary recommended
method for using Ollama.

Configuration:

Set llm-platform to ollama.
Set searchblox-llm to the base URL of your Ollama instance.
The openai-api-key is not used.

Example:

llm-platform: "ollama" 
searchblox-llm: "http://localhost:11434" 
openai-api-key: "" 
models: 
chat: "llama3" 
document-enrichment: "mistral" 
# ... other models

Llama.cpp

Use this provider to connect directly to a running llama.cpp server instance.

Configuration:

Set llm-platform to llamacpp.
Set searchblox-llm to the URL of your llama.cpp server.
The openai-api-key is not used.

Example:

llm-platform: "llamacpp" 
searchblox-llm: "http://localhost:8080" 
openai-api-key: "" 
models: 
chat: "your-local-gguf-model-name" 
document-enrichment: "your-local-gguf-model-name" 
# ... other models