- You can index only the sitemaps listed on the webpage by enabling sitemaps in WEB collection settings.
- We support the sitemap.xml if it is in the robots.txt when the settings ”Follow sitemap" is enabled.
- If the sitemap.xml has a list of other sitemaps.xml SearchBlox would index the same.
- Standard XML sitemaps are supported by SearchBlox. Sitemaps with compressed XML files with tar or gzip extensions are not currently supported in SearchBlox.
- Only sitemaps would be indexed on disabling ignore sitemaps. That is, the links available in
https://example.com/sitemap.xmlwould be indexed.
- By default, ignore sitemaps would be enabled in WEB collection settings.
- Multiple sitemaps can be indexed by providing multiple URLs in the root path.
- Allow/Disallow collection path rules can be considered while indexing sitemap URLs.
- Spider depth and other WEB settings are not applicable for sitemaps.
Updated 4 months ago