For site owners
About our crawler
If you found TurtleLabs-WebOS in your access logs, this page explains what it was doing.
What it does
Somebody asked us to audit your site, almost certainly you or someone who works with you. We fetch the page they gave us, follow internal links, and read up to twelve pages one at a time.
What it does not do
- It does not submit forms, log in, or follow anything that looks like an action.
- It does not run your JavaScript.
- It does not fetch images, fonts or media. Only the HTML.
- It does not store your page content beyond producing the report.
- It does not crawl continuously. One audit is one visit.
How to stop it
Disallow TurtleLabs-WebOS in your robots.txt and it will not come back. Blocking the user agent at your edge works too, and we will not attempt to disguise ourselves if you do.