Logging

Log File Types and Purposes

• SearchBlox generates multiple log files, each serving a distinct purpose to facilitate targeted troubleshooting:

  1. index.log Captures all crawler indexing operations, providing visibility into document processing workflows and potential indexing issues.
  2. query.log Records search queries submitted to the system, enabling analysis of user search patterns and query performance.
  3. status.log Contains critical system information, including startup processes, operational status, errors, and warnings.
  4. opensearch.log Documents OpenSearch activities, which powers SearchBlox's core search functionality.
  5. auth.log Tracks authentication events related to user logins in the administration dashboard.
  6. activity.log Records all collection-related operations, particularly creation and deletion activities.
  7. parameter. log stores parameter-related information and changes within the system.
  8. user.log Documents user management activities, including user additions, modifications, and deletions.
  9. securesearch.log Contains entries specific to secure search functionality and operations.
  10. pretext.log Records all pretext-related activities and processes.
  11. chatbot.log Captures interactions and operations related to chatbot functionality.

• Additionally, service-related activities are logged in: <SEARCHBLOX_INSTALLATION_PATH>/services/log/wrapper.log

Log Rotation

SearchBlox implements automatic log management to prevent disk space issues:

i. OpenSearch, status, index, query, and other logs undergo daily rotation.
ii. Log files are automatically purged after a fourteen-day retention period.

Disabling Logging

In certain scenarios, administrators may need to disable logging functionality. The process differs by the operating system

Windows:
Navigate to the service configuration file: <installation-folder>/service/service.xml
Locate the line containing the<log mode="roll-by-time">
Modify it to <log mode="OFF">

Linux:
Open the startup script: <installation-folder>/bin/start.sh
Locate the startup command and modify it to: eval java $JAVA_OPTS -jar start.jar \ &

Important: After making these configuration changes, SearchBlox must be stopped and started for the changes to take effect.

Error Messages Within the Logs

Critical Errors and Troubleshooting

The status.log and opensearch.log files can be monitored for the following errors:

1. gc overhead limit exceeded-Increase JVM memory allocation and restart the service.
2. OutOfMemoryError- Increase system or allocated memory and restart.
3. java.lang.OutOfMemoryError:Java heap space- Increase Java heap space in JVM settings.
4. java.lang.OutOfMemoryError: PermGen space- Increase system or allocated memory and restart.
5. shutdown or Shutdown- Investigate system resources and network connectivity.
6. high disk watermark [*] exceeded- Increase allocated disk space.
7. Too many open files- Increase ulimit values for the SearchBlox service user. Set it to maximum.

Troubleshooting

Opensearch log in Linux

Error specific to /opt/searchblox/opensearch/logs/searchblox.log for Linux:

• max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Resolution:
Increase map count using the command:
sysctl -w vm.max_map_count=262144

Too Many Open Files

If this error is observed in index.log or status.log it is required to increase the ulimit.

• Check the ulimit using the command:
  ulimit -n
• Edit the limits.conf file using the command:
  vi /etc/security/limits.conf
• Give the following in the file before the end of file

* soft nofile 100000
* hard nofile 100000

root soft nofile 100000
root hard nofile 100000

• Now you can check the ulimit value using the command:
  ulimit -n
• ulimit value should be 100000 or higher
• Please reboot after this change.

Monitoring OpenSearch Indices

SearchBlox ingests and retrieves data from OpenSearch. You can access OpenSearch directly to view this data.

You can check the status of OpenSearch indices through the web interface or command line:
For Web Interface Navigate to: https://localhost:9200/_cat/indices

Command Line (Linux)
Use the following curl command:
curl -k -u <opensearch-username>:<opensearch-password> https\://localhost:9200/\_cat/indices?pretty=true

The output will display important information about each index:
Health status (green, yellow, red)
Status (open, close)
Collection id(idx001)
Number of documents
Size on disk

🚧

Important Information:

URL for accessing opensearch is https://localhost:9200/
If you are using a custom domain or port in opensearch.yml please use the relevant domain and port to access opensearch.