Cluster Setup on AWS

Please setup the 3 individual SearchBlox Servers and apply the license keys to all servers before proceeding with the cluster setup instructions.

To create a 3-server SearchBlox Cluster on AWS EC2 (1 indexing server and 2 search servers), follow the step-by-step instructions.

Configure EC2 instances and SearchBlox

1. Create or edit an existing AWS EC2 security group for the three SearchBlox servers, with the following rules:

📘

Note

• sg-xxxxxxxx is the ID of the security group (Search is the name of the group); all instances started with this security group allow internal traffic.

2. Please create 3 servers (one Index server and two Search servers ) in Amazon EC2.

3. Install SearchBlox in all three instances using the instructions Installing on Amazon Linux 2

4. If you want more security and use the internal IP for communication between the servers you need not make the changes provided in the next 4 steps. If you want to proceed with internal IPs for Elasticsearch, please make sure that all three servers are able to access the other server's Elasticsearch using the internal IP.

5. If you want an external IP for Elasticsearch, please make sure to allocate static public IP to the EC2 instances through Elastic IP in AWS.

6. Go to /etc/hosts and allocate the public IP to public DNS name, as following and restart

34.xxx.xx.78  ec2-xx-xxx-3-78.compute-1.amazonaws.com
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost6 localhost6.localdomain6

7. Now Go to /etc/hostname and make sure that the public DNS name is given in the file

8. Now restart the instance and SearchBlox. Now the IP in SearchBlox Admin Console would be public IP. Please request a license key for this IP address.

Please make this change in all the servers within the cluster setup

9. If you are not making any changes then the internal IP address would be displayed on the Admin Console license page. If you are to use the same for a cluster then get the license for this IP address.

10. Access SearchBlox Admin Console and upload license in all servers.

11. Restart all 3 SearchBlox servers.

❗️

Security Requirements:

• Traffic must be allowed on ports 9200 and 9300 among these 3 servers.
• Disallow external traffic access to 9200 and 9300.
• Allow traffic port 8443 to access the SearchBlox service.
• Allow traffic port 8444 to access the SearchBlox Analytics service.

Configure Cluster Setup through Index Server Admin Console

1. Go to Dashboard -> Administration -> Cluster in Index Server as shown below:

2. Select Multi-Node as in the screenshot below:

3. Enter the IP addresses for Index and Search Servers to join the cluster through Index Server.

Indexing Server:
Enter the IP address of the current server from where you configure the cluster, which is to be the Index server.

Search Server:
Enter the IP addresses of the Search Server 1 and Search Search Server 2, which are to be the Search servers.

Cluster name: (Optional)
If required, you can provide a unique cluster name for your SearchBlox cluster setup.

4. If you are using the internal IP for Elasticsearch and license in SearchBlox you can skip this step.
   If you are using public IP for Elasticsearch then
   Stop SearchBlox
   go to <SEARCHBLOX_INSTALLATION_PATH>/elasticsearch/config/elasticsearch.yml file make the changes as shown below:

network.bind_host: 172.30.3.4
network.publish_host: 34.231.3.78

where network.bind_host value is the private IP and network.publish_host is the public IP which is now mapped in /etc/hosts.

...
searchblox.master: 34.231.3.78
network.bind_host: 172.30.3.4
network.publish_host: 34.231.3.78
discovery.zen.ping.unicast.hosts: [34.231.3.78, 52.22.187.131]

5. Stop and Start SearchBlox in all the three servers (which might take a few minutes).

🚧

Important Information on Restart:

• Be sure to stop and start all the servers after adding the cluster nodes and changing the cluster name by logging into the AWS server.

• To stop the service, use the following command:
  systemctl stop searchblox

• To start the service, use the following command:
  systemctl start searchblox

6. Login to the Index server, create a new collection and start indexing the collection. The new collection will now be available on the 2 other search servers for search within a few minutes if the setup is successful.

📘

Note:

Search Server's Admin Console will not be accessible after successful cluster setup. You can use base search page link https://<Search-Server-IP-or-DNS>:<Port>/search/index.html to access the collection documents.

Continue with the following troubleshooting steps if the new collection is not shown on the search servers

7. Run the following command on all servers to check the cluster status:
   curl -XGET 'https://<IP>:9200/_cluster/health?pretty=true'
   The url can also be accessed at https://<IP>:9200/_cluster/health?pretty=true

8. The cluster health status needs to be green or yellow

9. The number_of_nodes has to be 3 for a three server cluster setup. If the number_of_nodes is 3 and the cluster health is green or yellow, then the setup is successful.

10. Create more collections, configure and index the on the indexing server, the same collections will automatically get created in search servers and available for search access.

Indexing and Searching in Cluster Setup

• Indexing and Admin Console configuration is only allowed on the Index server.
• Search is possible from both Index and Search servers.
• If the Index server goes down, searching is still possible through search servers until the indexing node recovers.
• If a search server goes down, indexing is possible through Index server and searching is possible through both Index and Search Servers. When the failed search server comes up, the indexed data will be replicated and be in sync with other 2 servers.

Upgrade Cluster

A clean installation to cluster similar to standalone servers is recommended. Cluster setup is a one time configuration. If the cluster setup has to be reconfigured using the same servers, Admin console features 'Multi Node' to 'Single Node' selection will not work.

Best Practices for Cluster Setup

• Load balancer needs to be setup to distribute the load between search servers.
• Ensure searches are done only on the Search Servers for best performance.
• Ensure that Index and Search servers are able to connect with each other on the same network.
• Port 9200 and 9300 should be open across all the three servers. Traffic must be allowed on ports 9200 and 9300 among these 3 servers.
• Elasticsearch URLs should be accessible across all three servers
• Please restart all servers after adding the cluster nodes and/or changing the cluster name.
• After configuring cluster setup, we recommend not to choose Single Node setup as it requires additional manual steps to remove cluster setup.