SowhatlabsBot

What it is

SowhatlabsBot indexes web content, analyses site performance and gathers data used to produce insight for website owners, marketers and analysts. It helps us understand how a site is structured and where there is room to improve on SEO, content and technical performance.

How to identify it

Requests carry this user-agent:

Mozilla/5.0 (compatible; SowhatlabsBot/1.0; +https://sowhatlabs.com/bot)

If you see that string in your logs, it is us. The URL in it points back to this page so you can find out what it was doing without having to ask.

How it behaves

• It obeys robots.txt. The crawler only requests pages a site administrator has allowed.
• It crawls gently. Request rates are kept low enough not to be felt by the server.
• It identifies itself. No disguised user-agent, and no crawling from residential proxies.

Why it visits your site

• Analyse SEO performance. How well a site is optimised for search engines.
• Monitor content quality. Relevance, freshness and engagement potential.
• Enhance user experience. Technical issues affecting performance or usability.
• Provide competitive insight. Where our clients stand relative to their market.

How to control it

You decide what the crawler may access, using your robots.txt. To allow it everywhere:

User-agent: SowhatlabsBot
Disallow:

To block it entirely:

User-agent: SowhatlabsBot
Disallow: /

To keep it out of one section only:

User-agent: SowhatlabsBot
Disallow: /private/

Changes take effect the next time the crawler reads your robots.txt.

Questions

If the crawler is causing a problem, or you want it to stop and would rather tell us than edit a file, email support@sowhatlabs.com or use the contact form. We will act on it.