HTTP API
The SearchBlox HTTP-API enables you to index and search web content using simple HTTP POST and GET actions. The HTTP-API can add and delete HTTP collections, update paths and settings, schedule indexing and stop indexing the collection. SearchBlox HTTP-API provides methods for working with HTTP collections of REST requests with JSON payloads.
Adding a Collection
Create a new Collection. You can create HTTP, File and Database collections using this API.
##Index URL
https://localhost:8443/rest/collection/add
Method
POST
Media Type
application/json
Headers
content-type : application/json
accept: application/json
Headers
SB-PKEY
LmfxTTDSeYxHTntJMHuhwRrGVICMaVN/wl/zPuQ3LtQDNRMnng5GpKIkgt0q1rCC/h6wDA==
Document Syntax
Create HTTP Collection
{
"apikey" : "61282E82E5D6D8D409EFC87E8415CDAA",
"publiccol":true,
"encrypted":false,
"colname":"httpcollection",
"coltype":"http",
"language":"en"
}
Create File Collection
{
"apikey" : "61282E82E5D6D8D409EFC87E8415CDAA",
"publiccol":true,
"encrypted":false,
"colname":"filecollection",
"coltype":"file",
"language":"fr"
}
Create Database Collection
{
"apikey" : "61282E82E5D6D8D409EFC87E8415CDAA",
"publiccol":true,
"encrypted":false,
"colname":"dbcollection",
"coltype":"db",
"language":"de"
}
Document Description
JSON Fields | Value |
---|---|
apikey | API key accessible in the Dashboard->Administration->License. It is also present in the config.xml file. |
publiccol | Collection access type either true or false. If "true", then the access type is "public". If "false", then the access type is "private". |
encrypted | Collection encryption either true or false. |
colname | Name of the collection. |
coltype | Type of the collection. The values given for WEB, file and database collection are HTTP, file and db respectively. |
language | Language of the collection specified in two-letter code https://developer.searchblox.com/docs/supported-languages |
Response Codes
Response Code | Message |
---|---|
601 | API Key is not found/invalid |
50001 | Collection Already Exists Collection Type: <coltype> not found |
5000 | Collection <colname > created successfully with type <coltype> |
Deleting a Collection
You can delete collections using this API.
Index URL
https://localhost:8443/rest/collection/delete
Method
POST
Media Type
application/json
Headers
content-type : application/json
accept: application/json
Headers
SB-PKEY
LmfxTTDSeYxHTntJMHuhwRrGVICMaVN/wl/zPuQ3LtQDNRMnng5GpKIkgt0q1rCC/h6wDA==
Document Syntax
{
"apikey" : "61282E82E5D6D8D409EFC87E8415CDAA",
"colname":"httpcollection"
}
Document Description
JSON fields | apikey |
---|---|
apikey | API key accessible in the Dashboard->Administration->License. It is also present in the config.xml file. |
colname | Name of the collection. |
Update the Collection Path
You can update HTTP collection path settings to configure rooturls, allowpaths, disallowpaths and formats for indexing.
Index URL
https://localhost:8443/rest/collection/updatePath
Method
POST
Media Type
application/json
Headers
content-type : application/json
accept: application/json
Headers
SB-PKEY
LmfxTTDSeYxHTntJMHuhwRrGVICMaVN/wl/zPuQ3LtQDNRMnng5GpKIkgt0q1rCC/h6wDA==
Document Syntax
{
"apikey": "61282E82E5D6D8D409EFC87E8415CDAA",
"colname": "httpcollection",
"rooturls": [
"http://www.google.co.in",
"http://www.bing.com"
],
"allowpaths": [
".*"
],
"disallowpaths": [
"http://www.google.co.in/test/bingo"
],
"allowformat": ["HTML","PPT","PDF","RTF","WORD","TEXT","EXCEL","ODF","WP","JPG","TIFF","XML","PSD","MP3","MP4","FLV","DWG","VSD","KEY","PAGES","NUMBERS","EPUB","PNG","GIF","SVG","WAV","AIF","MIDI","MPEG","MID","MPG","AIFF","BMP"]
}
Document Description
JSON Fields | Value |
---|---|
apikey | API key accessible in the SearchBlox Admin Console. It is also present in the config.xml file. |
colname | Name of the collection. |
rooturls | The root URL is the starting URL for the spider. It requests this URL, indexes the content, and follows links from the URL. Make sure the root URL entered has regular HTML HREF links that the spider can follow. |
allowpaths | The allowpath limits the spider to stay only within the given path or list of paths. Example: https://www.searchblox.com/ (Informs the spider to stay only within the searchblox.com site). |
disallowpaths | The disallowpath is the path or list of paths that you do not want the spider to crawl or index. |
allowformats | Allowformats are the file types that are to be indexed. File types other than those specified here will not be indexed. |
Update the Collection Settings
Update HTTP collection settings where you can configure various parameters to filter indexing.
Index URL
https://localhost:8443/rest/collection/updateSettings
Method
POST
Media Type
application/json
Headers
content-type : application/json
accept: application/json
Headers
SB-PKEY
LmfxTTDSeYxHTntJMHuhwRrGVICMaVN/wl/zPuQ3LtQDNRMnng5GpKIkgt0q1rCC/h6wDA==
Document Syntax
{
"apikey": "61282E82E5D6D8D409EFC87E8415CDAA",
"colname": "httpcollection",
"keyword-in-context": "false",
"remove-duplicates": "false",
"boost": "100",
"stemming": "false",
"spelling": "true",
"logging": "true",
"faqContent":"false",
"html-settings": {
"description": "meta",
"max-doc-age": "-1",
"max-doc-size": "-1",
"spider-max-depth": "6",
"spider-max-delay": "1",
"user-agent": "SearchBlox",
"referer": "Google",
"ignore-robots": "false",
"follow-sitemap": "false",
"follow-redirect": "true"
},
"basic-auth-settings": {
"username": "searchblox",
"password": "testing"
},
"form-auth-settings": {
"form-url": "http://www.google.co.in",
"form-action": "post",
"form": [{
"name": "httpcollection",
"value": "google"
}, {
"name": "httpcollection1",
"value": "searchblox"
}]
},
"proxy-settings": {
"server-url": "http://searchblox.com/proxy",
"username": "proxy",
"password": "adasd"
}
}
Document Description
JSON Fields | Attributes | Value |
---|---|---|
apikey | API key accessible in the SearchBlox Admin Console. It is also present in the config.xml file. | |
colname | Name of the Collection. | |
keyword-in-context | Value is set to Yes or No to enable or disable keyword-in-context display respectively. The keyword-in-context returns search results with the description displayed from content areas where the search term occurs. To enable give yes and to disable give no. | |
remove-duplicates | Value is set to Yes or No to remove duplicates or allow duplicate documents while indexing respectively. | |
boost | Boost search terms for the collection by setting a value greater than 1 (maximum value 9999). | |
stemming | Value is set to Yes or No to enable or disable stemming respectively. When stemming is enabled, inflected words are reduced to root form. For example, "running", "runs", and "ran" are the inflected form of run. | |
logging | Value is set to Yes or No to enable or disable logging respectively. | |
faqContent | Value is set to Yes or No to enable or disable SmartFAQs content respectively. | |
html-settings | description | This description setting configures the HTML parser to read the description for a document. You can specify any one of the following HTML tags to be read as description. Description, h1, h2, h3, h4 ,h5, h6. |
max-doc-age | Specifies the maximum allowable age in days of a document in the collection. By giving -1 we do not specify any maximum allowable age. | |
max-doc-size | Specifies the maximum allowable size in kilobytes of a document in the collection. By giving -1 we do not specify any maximum document size. | |
spider-max depth | Specifies the maximum depth the spider is allowed to proceed to index documents. Value can be specified from 1-10. | |
spider-max-delay | Specifies the wait time in milliseconds for the spider between HTTP requests to a web server. By giving 0 we specify no delay. | |
user-agent | Specifies the name under which the spider requests documents from a webserver. | |
referer | Specifies the URL value set in the request headers to specify where the user agent previously visited. | |
ignore-robots | Value is set to Yes or No to tell the spider to obey robot rules or not. | |
follow-sitemap | Value is set to Yes or No to tell the spider whether sitemaps alone can be indexed, or if all of the URLs have to be indexed respectively. | |
follow-redirect | Is set to Yes or No to instruct the spider to automatically follow redirects or not. | |
basic-auth-settings | username | These settings help in indexing content secured by HTTP Basic authentication. Username for basic authentication has to be specified for this attribute. |
password | Password for basic authentication has to be specified for this attribute. | |
form-auth-settings | form-url | These settings help in indexing content protected using form based authentication. Form-url is the ACTION URL of the authentication HTML form. |
Form-action | Specifies whether the form action is a POST or GET. | |
form – name, value | The set of name/value pairs that are required. For example, username and password information for authentication are set here. Example Name,Value Web User,myself Password,abc123 Login,true | |
proxy-settings | server-url | These settings help in indexing content through proxy servers. This specifies the URL to access the proxy server. |
username | When the proxy server requires authentication, set the username. | |
password | Set the password. |
Update the Scheduler Settings
Index URL
https://localhost:8443/rest/collection/updateScheduler
Method
POST
Media Type
application/json
Headers
content-type : application/json
accept: application/json
Headers
SB-PKEY
LmfxTTDSeYxHTntJMHuhwRrGVICMaVN/wl/zPuQ3LtQDNRMnng5GpKIkgt0q1rCC/h6wDA==
Document Syntax
{
"apikey": "61282E82E5D6D8D409EFC87E8415CDAA",
"colname": "httpcollection",
"index":{
"frequency":"ONCE",
"timestamp":"21-04-2024 19:05:00"
},
"clear":{
"frequency":"MINUTELY",
"timestamp":"21-04-2024 18:05:00"
}
}
Document Description
XML Tag | Attribute | Value |
---|---|---|
apikey | API key accessible in the SearchBlox Admin Console. It is also present in the config.xml file. | |
colname | Name of the Collection | |
index | frequency | Specifies the frequency of indexing of web documents. The values can be ONCE, HOURLY, DAILY, EVERY 48 HOURS, EVERY 96 HOURS, WEEKLY and MONTHLY. |
timestamp | Specifies the timestamp when the indexing has to start. Example: 21-01-2016 19:05:00. | |
timestamp | Specifies the timestamp when the clearing has to occur. Example: 21-01-2016 19:05:00. |
Index or Stop Indexing the collection
Index URL
https://localhost:8443/rest/collection/actions
Method
POST
Media Type
application/json
Headers
content-type : application/json
accept: application/json
Headers
SB-PKEY
LmfxTTDSeYxHTntJMHuhwRrGVICMaVN/wl/zPuQ3LtQDNRMnng5GpKIkgt0q1rCC/h6wDA==
Document Syntax
Indexing collection
{
"apikey": "61282E82E5D6D8D409EFC87E8415CDAA",
"colname": "httpcollection",
"action":"index"
}
Stop Indexing Collection
{
"apikey": "61282E82E5D6D8D409EFC87E8415CDAA",
"colname": "httpcollection",
"action":"stop"
}
Document Description
JSON fields | Value |
---|---|
apikey | API key is accessible in the SearchBlox Admin Console. It is also present in the config.xml file. |
colname | Name of the collection. |
action | - Specifies the type of action to be performed. Index is specified to start indexing the collection. - If the indexing process is going on, stop is specified to stop indexing the collection. |
Response Code Description
5000 | Collection <colname > created successfully with type <coltype> |
50001 | Collection Type: <coltype> not found |
50002 | Invalid JSON |
50003 | Collection <colname> Deleted Successfully |
50005 | Collection path has been saved successfully |
50006 | Specified collection is not a CUSTOM collection |
50007 | Invalid Request/Collection Not Found |
50008 | Collection started indexing. Collection has been stopped. Collection schedule saved successfully |
50009 | Invalid Request/ Collection Not Found |
601 | API Key Not Valid |
Updated 5 months ago