SearchBlox

SearchBlox Developer Hub

Welcome to the SearchBlox developer hub. Here you will find comprehensive guides and documentation to help you start working with SearchBlox as quickly as possible, as well as support if you get stuck. Let's jump right in!

Documentation

SearchBlox for Amazon Elasticsearch Service

SearchBlox for Amazon Elasticsearch Service is an enterprise search platform for the AWS Cloud thats uses the Amazon Elasticsearch Service, the fully managed and scalable Elasticsearch service available on Amazon Web Services (AWS). SearchBlox for Amazon Elasticsearch Service can crawl, index and search content across multiple datasources including file systems, websites, databases and applications.

Architecture

SearchBlox for Amazon Elasticsearch Service consists of two types of SearchBlox servers that are available through the AWS marketplace. The first is SearchBlox IndexServer. The SearchBlox IndexServer can crawl and index content in over 40 document formats including PDFs, HTML and Microsoft Word, Excel, Powerpoint directly into Amazon Elasticsearch Service. The second type of server is the SearchBlox SearchServer. The SearchBlox SearchServer provides ready-to-use, fully customizable search front-ends including faceted search for the indexes created by the SearchBlox IndexServer in the Amazon Elasticsearch Service.

Setup

AWS Region

Please make sure to select same AWS Region in all the steps mentioned below. For example we have chosen "us-east-1" for creating elasticsearch, SearchBlox IndexServer , SearchBlox SearchServer etc.

1. Create VPC

Create a VPC which needs to be mentioned while creating a SearchBlox IndexServer at AWS Marketplace

2. Create KeyPair

Create a Key Pair and store it safely to access your AWS instance.

SSH

Use the keypair to ssh to AWS instance. If you are using Windows please use puttygen to convert the pem file to ppk file. Use this ppk file to connect to instance using putty.

3. Create IAM Role

Create an IAM role called SearchBlox_AmazonES with an AmazonESFullAccess Policy as shown in the screenshot. This role has to be configured after creating SearchBlox IndexServer (and search server if available) instance.

4. Create AWS Elasticsearch Domain

1.Give the Domain name and select Elasticsearch version 5.1

Elasticsearch Version

SearchBlox currently supports only Elasticsearch 5.1 on Amazon Elasticsearch Service

  1. Give the number of instances (between 1 and 20) and select the instance type as c4.xlarge.elasticsearch
  1. EBS Volume size can be set to 150GB or higher
  1. You can specify the start hour where Amazon AWS takes snapshot of cluster. Please specify the UTC time in the field
  1. You can specify access from to specific domain i.e., index and search servers by giving the private IPs of those servers. Select Allow access to the domain from the specific IP(s)
  1. Specify the comma separated IPs
  1. Review and create Elasticsearch domain.
    Elasticsearch Service Dashboard would have the domains created after 10 to 15 minutes.
  1. After configuring and connecting SearchBlox IndexServer (check the next section) On selecting a domain you can
    • View Cluster health
    • View Status of Indices
    • View the mappings of fields within the indices
    • Monitor the status of the elasticsearch service

5. Start SearchBlox IndexServer via Amazon Marketplace

Go to the AWS Marketplace https://aws.amazon.com/marketplace
Search for SearchBlox. Select IndexServer. For cluster setup, create SearchBlox SearchServer after creating SearchBlox IndexServer.

Check and click continue, you would go to the page below

Select the VPC created in earlier step

Select the Key Pair created earlier and launch the instance

Go to EC2 Dashboard

Integrate with IAM Role

This is an important step where we integrate IAM role with SearchBlox IndexServer .
Right Click the Server Instance, Go to Instance Settings -> Attach Replace IAM Role

Select and save the role to the instance

SSH into SearchBlox IndexServer

  • ssh into the SearchBlox IndexServer instance using the user ec2-user and the pem or ppk file.
    • Change user to jetty
sudo su - jetty
  • Edit /srv/jetty/sb/webapps/searchblox/WEB-INF/elasticsearch.yml to update the properties for AWS ES domain as follows:
searchblox.aws.region: us-east-1
searchblox.aws.url:  https://search-XXXXXX.us-east-1.es.amazonaws.com

The aws.region is the region selected while creating SearchBlox IndexServer and elasticsearch instance, the same would be also available in the aws url in elasticsearch.
The aws.url is the endpoint specified in the elasticsearch instance.

  • Restart SearchBlox as follows
service jetty restart

SearchBlox Basic Search url: https://xxxx:8443/searchblox/search.jsp
SearchBlox Faceted Search url : https://xxxx:8443/searchblox/plugin/index.html

where xxxx is is the Public DNS of the SearchBlox SearchServer instance

Kibana and Amazon Elasticsearch Service

  • Data indexed as well as logs are stored in elasticsearch domain. To view the logs the user can map the elasticsearch index named sbindexlog in kibana and search for the entries
    The Kibana link would be available in the Domain dashboard. Refer screenshot below:
  • Click the link and access Kibana.
  • Adding log indexes in Kibana. sbindexlog and sbstatuslog are the two logs that can be added in kibana. You can add the both logs in one index pattern

or create separate index pattern for each log

You can query the logs based on url, timestamp,etc

  • It is also possible to delete indexes via Kibana. Go to Dev Tools in left menu

Click Get to Work. You can delete the elasticsearch indices from here.

SearchBlox for Amazon Elasticsearch Service