SearchBlox for Amazon OpenSearch Service

SearchBlox for Amazon OpenSearch Service is an enterprise search platform for the AWS Cloud thats uses the Amazon OpenSearch Service, the fully managed and scalable Opensearch service available on Amazon Web Services (AWS). SearchBlox for Amazon OpenSearch Service can crawl, index and search content across multiple datasources including file systems, websites, databases and applications.

Architecture

This service consists of two types of SearchBlox servers that are available through the AWS marketplace. The first is SearchBlox IndexServer. The SearchBlox IndexServer can crawl and index content in over 40 document formats including PDFs, HTML and Microsoft Word, Excel, Powerpoint directly into Amazon OpenSearch Service. The second type of server is the SearchBlox SearchServer. The SearchBlox SearchServer provides ready-to-use, fully customizable search front-ends including faceted search for the indices created by the SearchBlox IndexServer in the Amazon OpenSearch Service.

Setup

❗️

AWS Region

Please make sure to select the same AWS Region in all the following steps mentioned. For example, we have chosen "us-east-1" for creating opensearch, SearchBlox IndexServer , SearchBlox SearchServer, etc.

1. Create VPC

Create a VPC, which needs to be mentioned while creating a SearchBlox IndexServer at AWS Marketplace.

2. Create KeyPair

Create a Key Pair, and store it safely to access your AWS instance.

❗️

SSH

Use the key pair to SSH to the AWS instance. If you are using Windows, use puttygen to convert the pem file to ppk file. Use this ppk file to connect to the instance using putty.

3. Create IAM Role

Create an IAM role called SearchBlox_AmazonES with an AmazonESFullAccess Policy, as shown in the screenshot. This role has to be configured after creating the SearchBlox IndexServer (and search server, if available) instance.

4. Create AWS Opensearch Domain

1. Select the Deployment Type and Opensearch Version 2.11, then click Next.

❗️

Opensearch Version

SearchBlox recommends the latest Opensearch Version 2.11 on Amazon OpenSearch Service.

2. Enter the domain name.

3. Select the Availability Zones, Instance Type and Number of Nodes.

4. Select the Storage Type, Size and also select the master node instance type and Number of Master Nodes.

5. Select the access and Security Type: VPC or Public access.

👍

Note

If you need any authentication select the Option below

6. Click Next to create Opensearch domain.

7. Opensearch Service Dashboard will have the domains created after 10 to 15 minutes.

8. Go to EC2 Service, click Launch Option, search for SearchBlox Enterprise Search AMI,
   and then click Create Instance.

5. Start SearchBlox IndexServer via Amazon Marketplace.

Go to the AWS Marketplace: https://aws.amazon.com/marketplace.
Search for SearchBlox and select IndexServer. For cluster setup, create SearchBlox SearchServer after creating SearchBlox IndexServer.

Check and click continue, which will take you to the following page:

Go to EC2 Dashboard.

Integrate with IAM Role

This is an important step where we integrate IAM role with SearchBlox IndexServer.
Right-click the Server Instance, then go to Instance Settings -> Attach Replace IAM Role.

Select and save the role to the instance.

SSH into SearchBlox IndexServer

• SSH into the SearchBlox IndexServer instance using the user ec2-user and the pem or ppk file.
  • Change user to SearchBlox.

sudo su - SearchBlox

• To update the Opensearch Domain Details on awssettings.yml file open the /opt/searchblox/webapps/ROOT/WEB-INF/awssettings.yml file and update the values and save the changes

The aws.region is the region selected while creating SearchBlox IndexServer and the Opensearch instance, which will also be available in the AWS URL in Opensearch. The aws.url is the endpoint specified in the Opensearch instance.

• To Update searchblox.yml
  /opt/searchblox/webapps/ROOT/WEB-INF/searchblox.yml and update the deployment type and
  save the changes.
  searchblox.deployment.type: aws

• Restart SearchBlox as follows:

• Access the SearchBlox Admin Console at https://xxxx:8443/console where xxxx is the Public DNS of the SearchBlox IndexServer instance.

• Access the SearchBlox Search URL as follows:

SearchBlox Search url : https://xxxx:8443/search

where xxxx is is the Public DNS of the SearchBlox SearchServer instance.

Increase RAM memory for SearchBlox in AWS.

After logging on as a jetty user using the following command:
sudo su - jetty
Go to edit /etc/default/jetty file and give the memory parameters in JAVA_OPTIONS. The content of the jetty file is given in the following:
12G refers to the 12 GB memory that has been allocated to SearchBlox

JAVA_OPTIONS="-server -Xms12G -Xmx12G -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djetty.http.host=0.0.0.0"
JETTY_HOME=/srv/jetty
JETTY_RUN=/srv/jetty/run
JETTY_USER=jetty
TMPDIR=/srv/jetty/temp
JETTY_BASE=/srv/jetty/sb

Kibana and Amazon OpenSearch Service

• Data indexed, as well as logs, are stored in the Opensearch domain. To view the logs, you can map the Opensearch index named sbindexlog in Kibana and search for the entries.
  The Kibana link will be available in the Domain dashboard. Refer to the following screenshot:

• Click the link and access Kibana.

• Adding log indices in Kibana.
  The two logs that can be added in Kibana are sbindexlog and sbstatuslog. You can add both logs in one index pattern.

Alternatively, you can create a separate index pattern for each log.

You can also query the logs based on URL, timestamp, etc.

• It is also possible to delete indices via Kibana. Go to Dev Tools in the left-hand menu. * To delete the Opensearch indices, click Get to Work .