OUR BOT
Shopsteddy respects robots.txt.
robots.txt is one of the ways a site owner tells automated crawlers where they may and may not go. We use the crawler name NBTBot.
Allow Shopsteddy
If you want Shopsteddy to be allowed across the public site, you can explicitly allow it:
User-agent: NBTBot
Allow: /Block Shopsteddy completely
If you don't want Shopsteddy to crawl the site, you can block it:
User-agent: NBTBot
Disallow: /Keep Shopsteddy away from selected paths
You can also block only the areas you don't want us to request:
User-agent: NBTBot
Disallow: /account/
Disallow: /private/What that means in practice
- We check robots.txt before ordinary crawling.We refresh it rather than assuming your rules never change.
- We follow the rules that apply to NBTBot.If you block a path for Shopsteddy, we leave that path alone.
- We keep the reason in the scan record.If robots.txt stops a check, the result says so instead of pretending the page failed.
- We don't work around your choice.We do not change bot identity or rotate around a block simply to reach content you told us not to crawl.