Browse Source

Block SemrushBot-BA from crawling the site

This seems to be a separate Semrush bot that doesn't listen to
robots.txt for the main SemrushBot UA. Their site also says to block
"SemrushBot-SA" and doesn't mention "-BA", but I don't know if that's a
mistake on their end or if this won't work.
merge-requests/85/head
Deimos 5 years ago
parent
commit
4377175a25
  1. 4
      tildes/static/robots.txt

4
tildes/static/robots.txt

@ -43,6 +43,10 @@ Disallow: /
User-agent: SemrushBot User-agent: SemrushBot
Disallow: / Disallow: /
# https://www.semrush.com/bot/
User-agent: SemrushBot-BA
Disallow: /
# https://www.zoominfo.com/ # https://www.zoominfo.com/
User-agent: ZoominfoBot User-agent: ZoominfoBot
Disallow: / Disallow: /
Loading…
Cancel
Save