A SearchBlox Cluster setup is designed to provide high availability, scalability, and fault tolerance for enterprise search operations. The cluster includes one Index Server and two Search Servers, each with specific roles to ensure continuous data indexing and efficient query handling.

Components of the SearchBlox Cluster

Index Server

• Role: Handles continuous crawling and data ingestion from multiple sources.
• Function: Synchronizes search indexes across all Search Servers.
• Benefit: Ensures the search index remains up to date with the latest data.

Search Servers (Search Server 1 and Search Server 2)

• Role: Process and distribute search queries.
• Function: Handle user search requests and return results quickly and reliably.
• Benefit: Load balancing between the servers ensures high availability and failover support.

SearchBlox Multi-Node Cluster

Setting up a Multi-Node Cluster in SearchBlox improves scalability, availability, and performance for enterprise search. The following steps explain how to configure the cluster directly from the SearchBlox Admin Console.

ℹ️

Recommended Cluster Configuration:

1 Index Server - Handles continuous crawling and indexing.
2 Search Servers - Distribute search queries and provide search results.

Prerequisites for Cluster Setup

1. Install SearchBlox:
   Install SearchBlox on three separate physical or virtual servers.

2. Network Connectivity:
   Ensure that all three servers can communicate with each other over the network. You should be able to ping each server using its IP address.

3. Port Configuration:
   Allow traffic on ports 9200 and 9300 between the three cluster servers. Make sure no firewall rules block these ports.

4. Delete Existing Collections:
   Delete all existing collections from all three servers before setting up the cluster.

5. Upload Server Licenses:
   Upload the appropriate license for each server. Save the IP address shown on the License page for every server.

Configure the Cluster from the Index Server Admin Console

1. Access the Index Server Admin Console:
   Log in to the Admin Console of the Index Server.

2. Navigate to Cluster Configuration:
   Go to Dashboard → Administration → Cluster.

3. Select Multi-Node Cluster:
   Select the Multi-Node option as shown in the screenshot.

3. Enter the IP addresses of the Index Server and Search Servers to add them to the cluster from the Index Server.
   

   3. Index Server:
      Action: Enter the IP address of the current server where you are configuring the cluster.
      Purpose: This server will act as the Index Server, responsible for continuous crawling and indexing.

   4. Search Servers:
      Action: Enter the IP addresses of Search Server 1 and Search Server 2.
      Purpose: These servers will handle search queries and return search results.

   5. Cluster Name (Optional):
      Action: Optionally enter a unique name for your SearchBlox cluster.
      Purpose: This helps identify the cluster if you manage multiple clusters.
      e cluster if you have multiple clusters.

Example Configuration:
Server Type IP Address
Index Server: 192.168.1.101
Search Server 1: 192.168.1.102
Search Server 2: 192.168.1.103
Cluster Name: MySearchBloxCluster

4. Save Configuration:
   After entering the details, save the configuration. The Admin Console will log out automatically after a few seconds.

5. Re-login and Create a New Collection:
   Re-login to the Index Server Admin Console.
   Create a new collection and start indexing. The new collection will be available on the two Search Servers within a few minutes if the setup is successful.

Accessing Search Servers:

After a successful cluster setup, the Admin Console of the Search Servers will not be accessible.
To access the search functionality, use the following URL

https\://<Search-Server-IP-or-DNS>/search/index.html

Troubleshooting Cluster Setup: If the new collection is not shown on the search servers

1. Check Cluster Status:

Run the following command on all servers to check the cluster status:

curl -XGET 'https\://<IP>:9200/\_cluster/health?pretty=true'

Alternatively, access the URL in a browser:

https\://<IP>:9200/\_cluster/health?pretty=true

2. Verify Cluster Health:
   Status: Should be green or yellow.
   Number of Nodes: Should be 3 for a three-server cluster setup.
   If the number_of_nodes is 3 and the cluster health is green or yellow, then the setup is successful.
3. Create More Collections:
   If the cluster status is correct, create additional collections on the Index Server. These collections will automatically be available on the Search Servers for search access.

Indexing and Searching in Cluster Setup

In a SearchBlox Multi-Node Cluster, the roles of the Index Server and Search Servers are clearly defined to ensure high availability, fault tolerance, and efficient search operations.

1. Indexing:
   1. Allowed Only on the Index Server:
      Indexing and Admin Console configuration (e.g., creating collections, configuring connectors, etc.) can only be performed on the Index Server.
   2. Data Replication:
      Once data is indexed on the Index Server, it is automatically replicated to the Search Servers for search availability.
2. Searching:
   1. Search from Both Index and Search Servers:
      Search queries can be executed from both the Index Server and the Search Servers.
   2. Load Balancing:
      Search queries are distributed between the Search Servers for optimal performance.
3. Fault Tolerance:
   1. If the Index Server Goes Down:
      Searching: Search queries can still be served by the Search Servers until the Index Server recovers.
      Indexing: Indexing operations will be paused until the Index Server is back online.
   2. If a Search Server Goes Down:
      Searching: Search queries can still be served by the Index Server and the remaining Search Server.
      Indexing: Indexing operations continue on the Index Server.
      Recovery: When the failed Search Server comes back online, it will automatically sync with the other servers to replicate the indexed data.
      
      

   Summary of Indexing and Searching in Cluster Setup

Manual Configuration of Cluster Setup

To manually configure a SearchBlox Multi-Node Cluster, follow these steps:

1. Install and Start SearchBlox Service:
   Install SearchBlox on all three servers (1 Index Server and 2 Search Servers). Start the SearchBlox service on each server. Import the license on each server. Ensure the License Screen IP matches the IP mapped in the license.xml file.
2. Check Port Connectivity:
   Verify that the required ports (9200 and 9300) are open and accessible between the servers.
   Use the telnet command to check connectivity:

telnet <IP or hostname> 9200
telnet <IP or hostname> 9300

If telnet is not available, use the curl command:

curl -k -u <username>:<password> https://<IP of cluster node>:9200
curl -k -u <username>:<password> https://<IP of cluster node>:9300

Ensure all SearchBlox Ports are allowed for internal network access.

3. Delete Default Collections:
   Delete the default HTTPCollection from the Manage Collections screen on all three servers.
   Stop the SearchBlox service on all servers. Ensure no other SearchBlox processes are running in the background.
4. Configure opensearch.yml:
   Add the following cluster configuration lines at the end of the opensearch.yml file on each server. Replace with the appropriate IP addresses:

cluster.name: <ClusterName>
node.name: <NodeName>
network.host: <IP>
discovery.seed_hosts: ["<IP1>", "<IP2>", "<IP3>"]
cluster.initial_master_nodes: ["<IP1>", "<IP2>", "<IP3>"]

Example Configuration:

cluster.name: MySearchBloxCluster
node.name: IndexServer
network.host: 192.168.1.101
discovery.seed_hosts: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]
cluster.initial_master_nodes: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]

• 'node.name' to be provided with the licensed IP address in the same format as shown in the reference.

• 'discovery.seed_hosts' will have all 3 cluster server IP addresses.

• 'cluster.initial_cluster_manager_nodes' will have all 3 cluster servers 'node.name' values.

5. Add the following line at the end of the searchblox.yml file in all 3 servers

searchblox.master: 172.16.1.247

• Here 'searchblox.master' IP represents Index server IP Address in all 3 searchblox.yml files.

6. Finally set the replicas count to 1 with the existing line in all 3 searchblox.yml file.

searchblox.replicas: 1

• Here 'searchblox.replicas' value represents the number of data copies that the cluster maintains over cluster servers.

6. Delete the nodes folder under <SEARCHBLOX_INSTALLATION_DIRECTORY>/opensearch/data/ path and stop SearchBlox. And also delete the logs folder files under <SEARCHBLOX_INSTALLATION_DIRECTORY>/opensearch/logs/

This is the step you need to follow when you face 503 error while setting up the cluster and starting the SearchBlox service.

7. Start Index Server followed by Search Server 1 and Search Server 2.

Troubleshooting:

If the Index Server does not start, try starting Search Server 1 and Search Server 2 first. This process will help discover the master node.After the master node is discovered, stop the SearchBlox servers that failed to start in the cluster. Then delete the nodes folder from the following path .../opensearch/data/`Restart SearchBlox. All three cluster SearchBlox servers should start successfully .Next, create a collection from the Index Server and perform searches from Search Server 1 and Search Server 2 to confirm that data replication is working correctly.

Upgrade Cluster

A clean installation for the cluster, similar to standalone server installation, is recommended. Cluster configuration is a one-time setup.

If you need to reconfigure the cluster using the same servers, switching from Multi Node to Single Node in the Admin Console will not work.

Best Practices for Cluster Setup

• Configure a load balancer to distribute traffic between the Search Servers.
• Perform searches only on the Search Servers for optimal performance.
• Ensure the Index Server and Search Servers can communicate with each other on the same network.
• Ports 9200 and 9300 must be open on all three servers, and traffic should be allowed between them.
• Ensure the OpenSearch URLs are accessible from all three servers.
• Restart all servers after adding cluster nodes or changing the cluster name.
• After configuring the cluster, avoid switching to Single Node mode, as removing the cluster configuration requires additional manual steps.