Logging

Log File Types and Purposes

• SearchBlox generates multiple log files, each serving a specific purpose for troubleshooting. These log files are located in <SEARCHBLOX_INSTALLATION_PATH>/webapps/ROOT/logs.

1. index.log records all crawler indexing operations and helps identify document processing or indexing issues.
2. query.log records search queries submitted to the system and helps analyze user search behavior and query performance.
3. status.log contains important system information such as startup processes, system status, errors, and warnings.
4. opensearch.log records OpenSearch activities that support SearchBlox search functionality.
5. auth.log records authentication events related to admin dashboard logins.
6. activity.log records collection-related operations, including creation and deletion.
7. parameter.log records parameter-related information and system configuration changes.
8. user.log records user management activities such as adding, modifying, and deleting users.
9. securesearch.log records events related to secure search functionality.
10. pretext.log records activities related to PreText processing.
11. chatbot.log records chatbot interactions and related operations.

• Service-related activities are logged in:
  `<SEARCHBLOX_INSTALLATION_PATH>/services/log/wrapper.lo

Log Rotation

SearchBlox implements automatic log management to prevent disk space issues.

• OpenSearch, status, index, query, and other logs undergo daily rotation.
• Log files are automatically purged after a fourteen-day retention period.

Disabling Logging

In certain scenarios, administrators may need to disable logging functionality. The process differs by the operating system.

Windows:
Navigate to the service configuration file: <installation-folder>/service/service.xml
Locate the line containing the<log mode="roll-by-time">
Modify it to <log mode="OFF">

Linux:
Open the startup script: <installation-folder>/bin/start.sh
Locate the startup command and modify it to: eval java $JAVA_OPTS -jar start.jar \ &

Important: After making these configuration changes, SearchBlox must be stopped and started for the changes to take effect.

Error Messages Within the Logs

Critical Errors and Troubleshooting

The status.log and es.log files can be monitored for the following errors:

1. gc overhead limit exceeded – Increase the JVM memory allocation and restart the service.
2. OutOfMemoryError – Increase the system memory or allocated memory and restart the service.
3. java.lang.OutOfMemoryError: Java heap space – Increase the Java heap space in the JVM settings.
4. java.lang.OutOfMemoryError: PermGen space – Increase the system memory or allocated memory and restart the service.
5. shutdown or Shutdown – Check system resources and network connectivity.
6. high disk watermark [] exceeded* – Increase the allocated disk space.
7. Too many open files – Increase the ulimit values for the SearchBlox service user and set them to the maximum.

Troubleshooting

Opensearch log in Linux

Error specific to /opt/searchblox/opensearch/logs/searchblox.log for Linux:

• max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Resolution:
Increase map count using the command:
sysctl -w vm.max_map_count=262144

Too Many Open Files

If this error is observed in index.log or status.log:

• Check the ulimit using the command:
  ulimit -n
• Edit the limits.conf file using the command:
  vi /etc/security/limits.conf
• Give the following in the file before the end of file

* soft nofile 100000
* hard nofile 100000

root soft nofile 100000
root hard nofile 100000

• Now you can check the ulimit value using the command:
  ulimit -n
• ulimit value should be 100000 or higher
• Please reboot after this change.

Monitoring OpenSearch Indices

SearchBlox stores and retrieves all search data through OpenSearch indices. Regularly checking the status of these indices helps ensure that data is being indexed correctly and that all collections are healthy and available.

You can view the index status using either a browser-based web interface or a command line tool use whichever method is most convenient for your environment:
For Web Interface Navigate to: https://localhost:9200/_cat/indices

Command Line (Linux)
Use the following curl command: curl -k -u <opensearch-username>:<opensearch-password> https://localhost:9200/_cat/indices?pretty=true

The output will show a row for each OpenSearch index with the following key pieces of information that help you assess its health and usage:
Health status (green, yellow, red)
Status (open, close)
Collection id(idx001)
Number of documents
Size on disk

🚧

Important Information:

URL for accessing opensearch is https://localhost:9200/
If you are using a custom domain or port in opensearch.yml please use the relevant domain and port to access opensearch.