DownloadsDocumentationAPI ReferenceChangelogOpen A TicketContact
Guides

Cluster Setup

Suggest Edits

SearchBlox Cluster(Multi Node Cluster) setup includes an Index Server and 2 Search Servers. The Multi Node Cluster setup provides the ability for continuous crawling/ingestion of data in the Index Server and the ability to distribute the search queries between Search Server 1 and Search Server 2. The load balanced Search Servers will provide an always on, secure, fast and fail-safe search service.

SearchBlox Multi Node Cluster

Multi Node Cluster set up can be done directly from the SearchBlox Admin Console. We recommend 3 SearchBlox Servers ( 1 Index and 2 Search servers) for the cluster setup.

Prerequisites for Cluster Setup

  1. Install SearchBlox on three different physical or virtual servers.

  2. Confirm that the three servers can see each other on the network, i.e., you should be able to ping the other servers using their IP.

  3. Traffic must be allowed on ports 9200 and 9300 among these 3 cluster servers without any firewall restrictions.

  4. Delete all existing collections in all three servers before configuring the cluster setup.

  5. Upload server licenses in all three servers. Save the IP addresses from license page for all three servers.

Configure Cluster Setup through Index Server Admin Console

  1. Go to Dashboard -> Administration -> Cluster in Index Server as shown below:

  1. Select Multi-Node as in the screenshot below:

  1. Enter the IP addresses for Index and Search Servers to join the cluster through Index Server.

Indexing Server:
Enter the IP address of the current server from where you configure the cluster, which is to be the Index server.

Search Server:
Enter the IP addresses of the Search Server 1 and Search Search Server 2, which are to be the Search servers.

Cluster name: (Optional)
If required, you can provide a unique cluster name for your SearchBlox cluster setup.

1575

  1. Admin console will get logged out after a few seconds.

  2. Please re-log into the Index server, create a new collection and start indexing the collection. The new collection will now be available on the 2 other search servers for search within a few minutes if the setup is successful. Please note that Search Server's Admin Console will not be accessible after a successful cluster setup. You can access https://<Search-Server-IP-or-DNS>/search/index.html page to access the collection documents.

Continue with the following troubleshooting steps if the new collection is not shown on the search servers

  1. Run the following command on all servers to check the cluster status:
    curl -XGET 'https://<IP>:9200/_cluster/health?pretty=true'

The url can also be accessed at https://<IP>:9200/_cluster/health?pretty=true

  1. The cluster health status needs to be green or yellow

  2. The number_of_nodes has to be 3 for a three server cluster setup. If the number_of_nodes is 3 and the cluster health is green or yellow, then the setup is successful.

  3. Create more collections, configure and index those collections on the indexing server, the same collections will automatically get created in search servers and available for search access.

Indexing and Searching in Cluster Setup

  • Indexing and Admin Console configuration is only allowed on the Index server.
  • Search is possible from both Index and Search servers.
  • If the Index server goes down, searching is still possible through search servers until the indexing node recovers.
  • If a search server goes down, indexing is possible through the Index server and searching is possible through both Index and Search Servers. When the failed search server comes up, the indexed data will be replicated and be in sync with the other 2 servers.

Manual Configuration of Cluster Setup

  1. Once the SearchBlox service is installed in all 3 cluster servers, start SearchBlox service in each server and import the license based on the License screen shown IP.
    Note: License screen IP should match with the IP mapped in the license.xml file.

  2. As a pre-requisite, please check the ports connectivity using the telnet command (telnet <IP or hostname> <Port Number>). Here port numbers to be checked for the cluster are 9200 and 9300. We also recommend you to check for SearchBlox Ports to be allowed for internal network access.

If telnet cannot be installed in your environment, you can use the below curl command using the license screen IP address of all 3 servers while the SearchBlox process is running:

curl -k -u <username>:<password> https://<IP of cluster node>:9200
curl -k -u <username>:<password> https://<IP of cluster node>:9300

  1. Now, delete the default HTTPCollection from Manage Collections Screen in all 3 servers and stop SearchBlox service(Ensure there is no other SearchBlox process is running in the background)

  2. Add the following cluster configuration lines at the end of the opensearch.yml file in each cluster node or server. Replace with the proper IP Address.

cluster.name: opensearch
network.bind_host: <IP>

node.name: sb_node_<IP>
discovery.seed_hosts: ["<IP>","<IP>","<IP>"]
cluster.initial_cluster_manager_nodes: ["<node.name>","sb_node_<IP>","<node.name>"]

  • 'node.name' to be provided with the licensed IP address in the same format as shown in the reference.

  • 'discovery.seed_hosts' will have all 3 cluster server IP addresses.

  • 'cluster.initial_cluster_manager_nodes' will have all 3 cluster servers 'node.name' values.

  1. Add the following line at the end of the searchblox.yml file in all 3 servers
searchblox.master: 172.16.1.247
  • Here 'searchblox.master' IP represents Index server IP Address in all 3 searchblox.yml files.
  1. Finally set the replicas count to 1 with the existing line in all 3 searchblox.yml file.

searchblox.replicas: 1

  • Here 'searchblox.replicas' value represents the number of data copies that the cluster maintains over cluster servers.

  1. Delete the nodes folder under <SEARCHBLOX_INSTALLATION_DIRECTORY>/opensearch/data/ path and stop SearchBlox. And also delete the logs folder files under <SEARCHBLOX_INSTALLATION_DIRECTORY>/opensearch/logs/

This is the step you need to follow when you face 503 error while setting up the cluster and starting the SearchBlox service.

  1. Start Index Server followed by Search Server 1 and Search Server 2.

Troubleshooting Step: When you dont find the Index server to start, try starting Search Server 1 and Search Server 2. In this process Master node will get discovered. And you can stop the SearchBlox Server which are failed to start from the Cluster. Delete the nodes folder from .../opensearch/data/ path. And Start SearchBlox. You should be able to start all 3 cluster SearchBlox Servers. Now create the collection through Index server and access Search from Search 1 and Search 2 servers to make sure data replication works.

Upgrade Cluster

A clean installation to cluster similar to standalone servers is recommended. Cluster setup is a one time configuration. If the cluster setup has to be reconfigured using the same servers, Admin console features 'Multi Node' to 'Single Node' selection will not work.

Best Practices for Cluster Setup

  • Load balancer needs to be setup to distribute the load between search servers.
  • Ensure searches are done only on the Search Servers for best performance.
  • Ensure that Index and Search servers are able to connect with each other on the same network.
  • Port 9200 and 9300 should be open across all the three servers. Traffic must be allowed on ports 9200 and 9300 among these 3 servers.
  • Elasticsearch URLs should be accessible across all three servers
  • Please restart all servers after adding the cluster nodes and/or changing the cluster name.
  • After configuring cluster setup, we recommend not to choose Single Node setup as it requires additional manual steps to remove cluster setup.

Updated 8 months ago

What’s Next

You can go through the related sections below