Logging

Logs

• SearchBlox provides multiple logs for troubleshooting and monitoring. The log files are available in <SEARCHBLOX_INSTALLATION_PATH>/webapps/ROOT/logs

  1. index.log (crawler indexing activities)
  2. query.log (search queries)
  3. status.log (startup, status, errors, warnings and license/IP address issues)
  4. es.log (elasticsearch activities)
  5. auth.log (user login in admin dashboard)
  6. activity.log (creation- and deletion-related activities for collections)
  7. parameter.log (parameters)
  8. user.log (users added/deleted/modified)
  9. securesearch.log (logs related to secure search)
  10. pretext.log (logs related to pretext activities)

• Elasticsearch and status logs are rotated daily and are not deleted.

• Index, query, and other logs are rotated daily and deleted after every fourteen days.

• SearchBlox service-related activities can be found the wrapper.log in ../services/log

• To turn off logging for searchblox.log follow the below configuration:

In Windows: Go to <installation-folder>/service/service.xml file. Edit the below configuration by updating <log mode="roll-by-time"> to **<log mode="OFF">**

In Linux: Go to <installation-folder>/bin/start.sh file and update below command as shown:
**eval java $JAVA_OPTS -jar start.jar \ &**

Note: Once you make the change please stop and start SearchBlox to take effect.

Log Files for Monitoring

The following log files are recommended for monitoring:

<SEARCHBLOX_INSTALLATION_PATH>/webapps/ROOT/logs/status.log
<SEARCHBLOX_INSTALLATION_PATH>/webapps/ROOT/logs/es.log

Linux path
/opt/searchblox/webapps/ROOT/logs

Windows path
C:/SearchbloxServer/webapps/ROOT/logs

Error Messages Within the Logs

Terms/errors to be monitored in the application log files

The status.log and es.log files can be monitored for the following errors:

1. gc overhead limit exceeded (memory issue)
2. OutOfMemoryError
3. java.lang.OutOfMemoryError: Java heap space
4. java.lang.OutOfMemoryError: PermGen space
5. shutdown or Shutdown
6. high disk watermark [*] exceeded
7. Too many open files

Possible Causes

• Errors 1 - 4 are due to insufficient RAM. Please increase memory allocation and restart.
• Error 5 indicates shutdown (reasons could be memory or disk issue or network issue)
• Error 6 indices insufficient disk space issue. Please increase disk space allocated.
• Error 7 occurs in Linux if the ulimit value is not set to maximum.

Troubleshooting

Elasticsearch log in Linux

Error specific to /opt/searchblox/elasticsearch/logs/searchblox.log for Linux:

• max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Resolution:
Increase map count using the command:
sysctl -w vm.max_map_count=262144

Too Many Open Files

If this error is observed in index.log or status.log it is required to increase the ulimit.

• Check the ulimit using the command:
  ulimit -n
• Edit the limits.conf file using the command:
  vi /etc/security/limits.conf
• Give the following in the file before the end of file

* soft nofile 100000
* hard nofile 100000

root soft nofile 100000
root hard nofile 100000

• Now you can check the ulimit value using the command:
  ulimit -n
• ulimit value should be 100000 or higher
• Please reboot after this change.

Elasticsearch Indices

SearchBlox ingests and retrieves data from elasticsearch, which you can access to view the data.

To check the status of elasticsearch index, go to https://localhost:9200/_cat/indices.

Alternatively, if you use Linux versions of SearchBlox you can use the following curl command

        `curl -k -u <elasticsearch-username>:<elasticsearch-password> https://localhost:9200/_cat/indices?pretty=true`

🚧

Important Information:

URL for accessing elasticsearch is https://localhost:9200/
If you are using a custom domain or port in elasticsearch.yml please use the relevant domain and port to access elasticsearch.