Amazon S3 Connector

Configuring SearchBlox

Before using the Amazon S3 Connector, install SearchBlox successfully, then create a Custom Collection.

Configuring Amazon S3 Connector

Prerequisites

All the files related to the connector should be available in the same folder that is, all files should be extracted into the same folder.
Create a data folder on your drive where the files would be temporarily stored and mention in yml files.

Please contact [email protected] to request the download link for Amazon S3 connector.

Steps to Configure and Run Amazon S3 Connector

In Windows, the files related to connector can be installed in the C:\ drive in Windows, in Linux it can be installed in /opt

Download the SearchBlox Amazon S3 connector. Extract the downloaded zip to a folder.
Unzip the archive under C:* or /opt*. Please find the screenshot on the extracted files in the following:

The extracted files would be an executable based on your OS, yml file and an encrypter.
Using the encrypter you can encrypt your access key and secret key from Amazon S3.

Configure the amazonS3.yml file which includes Amazon S3 properties and SearchBlox properties as listed in the following:


encrypted-ak	Encrypted access key from Amazon s3. Encrypter would be available in the downloaded archive
encrypted-sk	Encrypted secret key from Amazon s3. Encrypter would be available in the downloaded archive
region	Region of Amazon S3 instance
data-directory	Data Folder where the data needs to be stored. Make sure it has write permission.
api-key	SearchBlox API Key
colname	The name of the custom collection in SearchBlox.
queuename	Amazon S3 SQS parameter. This is required for update of documents after indexing in a bucket.
private-buckets	If set to true the content from private buckets will be indexed
public-buckets	If set to true the content from public buckets will be indexed
url	SearchBlox URL
includebucket	S3 buckets to be included
include-formats	File formats to be included
expiring-url	Expiring URLs in the search result. Default URL will be expiring URL
expire-time	Default expire time will be 300 mins
permanent-url:	permanent URLs in the search result. Among the expiring and permanent URL expiring URL is given more preference
max-folder-size	Maximum size of static folder after which it should be sweeped in MB.
servlet url & delete-api-url:	Make sure that the port number is right. If your SearchBlox runs in 8080 port the URLs should be right.

The content details of amazonS3.yml are provided here:

#User credentials
encrypted-ak: oSMMs-K1nkdjukjkk003kdkdW004k9GB0wUXjJoCkZ
encrypted-sk: 8kZf1ZPV_WcU9LBEJFdddIskdFzg5i6kNe-vk6ahffIvc8=
region: us-east-1
#Data Folder where the data needs to be stored Make sure it has write permission
data-directory: D:\GoWorkspace\searchblox\src\sbgoclient\examples\amazonS3
#SearchBlox API Key
api-key: 83F9C9AF71B1D0B334A7DDE36C99BF6A
#The name of the collection
colname: amazon
queuename: [testabcde]
#private-buckets if set to true will index files in private buckets else if set to false the private bucket files will not be indexed
private-buckets: true
#public-buckets if set to true will index files in public buckets else if set to false the public bucket files will not be indexed
public-buckets: false
#SearchBlox URL
url: http://localhost:8080/searchblox/rest/v2/api/
#IncludeBucket
includebucket: [tests3]
#The Included formats wont be indexed
include-formats: [.pdf,.docx,.xml,.json]
#servlet-url
servlet-url: http://localhost:8080/searchblox/servlet/SearchServlet
#expiring urls in search result. Default url will be expiring url and default expire time will be 300 mins
expiring-url: true
#expire time in min
expire-time: 300
#permanent urls in search result. Among the expiring and permanent url expiring url is given more preference
permanent-url: true 
#maximum size of static folder aftre which it should be sweeped in MB
max-folder-size: 2
#delete-api-url: http://localhost:8080/searchblox/api/rest/docdelete
delete-api-url: http://localhost:8080/searchblox/rest/v1/api/docdelete

Please make sure to give the API key, collection name, data folder path and Amazon S3 keys(encrypted) and other related properties.
Start running the amazonS3.exe file for Windows and ./amazonS3Linux32 or ./amazonS3Linux64

Steps to trigger notification when a document is updated in S3

In Amazon S3 console under services, click on the Simple Queue Service(SQS) which can be found under application integration.
The Select Create New Queue, then give an appropriate queue name and configure the queue.

Set message visibility timeout, retention days and receive message wait time.

Receive message wait time should be set between 1 - 20 seconds. So as to set the queue for long polling.
Then in the S3 console select a bucket whose SQS you need to set and click on “Properties”, Then in “Advanced settings” click on events.

Now select “Add Notification” and give a name to the notification and check the events for which you need notification.https://files.readme.io/e00dec7-4.png

Then in “Send To” fill SQS Queue and under SQS fill in the name of the queue which we created before and click on Save.

Now, whenever there is change to the documents in this particular bucket then a notification will be sent to SQS.
The queue names need to be given in the S3 connector yml file in order to trigger indexing whenever a document is updated