SearchBlox comes with a built-in web crawler that can collect and index information from different sources, including websites, intranets, and portals. It can automatically handle HTTPS content without special setup and is also able to crawl through a proxy server. In addition, it supports both HTTP Basic Authentication and Form-Based Authentication, allowing it to access and index protected or login-required pages securely.

Creating a Web Collection

To create a Web Collection in SearchBlox, follow these steps:

1. Log in to the Admin Console:

   • Start by signing in to your SearchBlox Admin Console using your administrator credentials.

2. Go to the Collections tab:

   • In the dashboard, open the Collections section.
   • Click “Create a New Collection” or simply press the “+” button to add a new collection.

3. Select the Collection Type

   • From the list of available collection types, choose “WEB Collection” since you want to index content from a website or intranet.

4. Configure the collection details

   • Name the collection: Provide a meaningful and unique name, such as “Intranet Site”, so it’s easy to identify later.
   • Configure RAG settings: Enable RAG if you want this collection to support ChatBot features or Hybrid RAG search.
   • Set access level: Choose Private if only specific users should access this collection, or Public if it should be available to everyone.
   • Enable encryption: Turn on Collection Encryption if your organization requires secure storage of indexed data.
   • Select language: Choose the appropriate content language if the data is not in English.

5. Save the collection

   • After confirming all settings, click Save to create your new collection and proceed with further configuration or crawling.

• Once the WEB collection is created you will be redirected to the Paths tab.

WEB Collection Paths

The WEB collection Paths allow you to configure the Root URLs and the Allow/Disallow paths for the crawler. To access the paths for the WEB collection, click on the collection name in the Collections list.

Root URLs

• The root URL is the main starting point for the crawler. It begins indexing from this page and follows the links found on it.
• Ensure the root URL has standard HTML HREF links so the crawler can move through the site properly.
• In the Paths sub-tab, add at least one root URL in the Root URLs field for the WEB Collection to start crawling field.

Allow/Disallow Paths

• Allow/Disallow paths tell the crawler which URLs to include or skip.
• They help manage a collection by removing unnecessary or irrelevant pages from indexing.
• In a WEB collection, at least one Allow path is required so the crawler stays within the correct subdomain defined in the Root URLs.

---

---

❗️

Important Note:

• Enter the Root URL domain name(s) (for example cnn.com or nytimes.com) within the Allow Paths to ensure the crawler stays within the required domains.
• If .* is left as the value within the Allow Paths, the crawler will go to any external domain and index the web pages.

WEB Collection Settings

The Settings page has configurable parameters for the crawler. SearchBlox provides default parameters when a new collection is created. Most crawler settings can be changed to meet your specific requirements.

📘

Generate Title, Description and Topics using SearchAI PrivateLLM and Enable Hybrid Search:

• Choose and enable Generate Using LLM and Auto Relevance

• Once you Enable Title/Description/Topics, we can choose Document Formats on which we ca generate it and also Enable Approval Option as shown.

  

• Enable Approval, if some URL needs user approval to generate Title/Descriptions/Topics. Enable and provide URLs which needs approval of a user.

  

• After indexing, user can see the Approval button for those URLs in Management Tab.

  

• By clicking Compare Keyword Search with Hybrid link, will redirect to the Comparison Plugin

Settings	Description
Title	Generates concise and relevant titles for the indexed documents using LLM.
Description	Generates the description for indexed documents using LLM.
Topic	Generates relevant topics for indexed documents using LLM based on document's content.
Auto Relevance	Enable/Disable Hybrid Search for automatic relevance ranking

📘

Process/Store Images Inside a Document Using SearchAI PrivateLLM:

• Enable Images, by default it is disabled.

• Choose the file formats where the images are located, currently SearchBlox supports WORD, PDF and PPT formats

• Enable Store images to store the processed images inside opensearch, by default it is disabled and stores the images locally inside SearchBlox installation path.
  NOTE: Indexing will be slower if it is enabled. So enable only if it is needed.

  

• While indexing based on the image, LLM will generate title and description for the processed image.

Settings	Description
Images	Enable to Generate Image's title and descriptions using LLM while indexing, by default it is disabled
Allowed Formats	Choose the file formats, currently SearchBlox supports WORD, PDF and PPT.
Store Images	Enable to store the processed images inside opensearch, by default it is disabled and stores the images locally inside SearchBlox installation path.

Synonyms

Synonyms help the search show relevant documents even when the exact search word is not used.
For example, if someone searches for “global,” the results can also include documents that use “world” or “international.”
We have an option to load Synonyms from the existing documents.

Schedule and Index

Sets the frequency and the start date/time for indexing a collection, from the root URLs. Schedule Frequency supported in SearchBlox is as follows:

• Once
• Hourly
• Daily
• Every 48 Hours
• Every 96 Hours
• Weekly
• Monthly

The following operation can be performed in WEB collections

Manage Documents

• Using Manage Documents tab we can do the following operations:

  1. Add/Update
  2. Filter
  3. View content
  4. View meta data
  5. Refresh
  6. Delete

• To add a document click on "+" icon as shown in the screenshot.

• Enter the document/URL, Click on add/update.

• Once the document is updated you will be able to see the document URL on the screen and we will be able to perform the above mentioned operations.

• 

Data Fields

Using Data Fields tab we can create custom fields for search and we can see the Default Data Fields with non-encrypted collection. SearchBlox supports 4 types of Data Fields as listed below:

1. Keyword
2. Number
3. Date
4. Text

📘

Note:

Once the Data fields are configured, collection must be cleared and re-indexed to take effect.

• The Data Fields tab is disabled while creating a collection. After the collection is created, you can see and edit the collection’s Data Fields from the Collection Edit page.
• When Default Data Fields are enabled, you will see SearchBlox’s built-in reserved fields. These reserved field names cannot be used for creating custom Data Fields.

• To create a custom Data Field, click the Add Data Field button, enter the field name from your document, choose the type of value it will store, and save it. The newly added Data Fields will then appear in the Data Fields list.

• Custom Data Fields can be shown in the Data Fields Tab as well as with the Default Data Fields list.

• Custom fields will also be shown under Deafult Data Fields list as shown below:

• Custom mapping file can be shown under the path: /webapps/ROOT/WEB-INF/mappings/collections

• We can edit the mapping file if we need to add any specific Analyzers. Mapping fields are added at the end of the collection mapping file as shown below:

📘

Note:

SearchBlox version 10.0.1 and above includes ML fields (ml_topic, ml_sentimentLabel, ml_entity_org, ml_entity_product, ml_entity_person, ml_entity_loc, ml_entity_gpe) as default mappings for all type of collections.

❗️

Important Note:

• Data Fields tab cannot be used with Encrypted type Collection.
• We recommend not to edit the collection mapping files unlesss we need to add analyzers or fielddata configuration.

Prompts

• When LLM/RAG is enabled, you can edit AI-based prompts for Title, Description, Topic, Image Description, and Smart FAQs.
• You can customize these prompts anytime, and use Restore Default to reset them back to the original SearchBlox settings.

---

• 

Models

The Models page allows you to configure and override AI models used for embeddings, reranking, and LLM-based features within the collection.

Embedding

• Provider specifies the embedding provider used to generate vector representations of documents.
• Model defines the embedding model used to convert document content into vectors for semantic search.

Reranker

• Provider specifies the reranker provider used for improving search result relevance.
• Model defines the reranker model used to re-score and reorder search results based on relevance.

LLM

• Provider specifies the Large Language Model provider used for AI-powered features.

• Model defines the LLM used for tasks such as document enrichment, summaries, and SmartFAQs.

• These settings override global configurations and apply only to the current collection.