From 4377175a25e6822d06665c4086ca046ca6c22715 Mon Sep 17 00:00:00 2001 From: Deimos Date: Mon, 23 Sep 2019 17:18:11 -0600 Subject: [PATCH] Block SemrushBot-BA from crawling the site This seems to be a separate Semrush bot that doesn't listen to robots.txt for the main SemrushBot UA. Their site also says to block "SemrushBot-SA" and doesn't mention "-BA", but I don't know if that's a mistake on their end or if this won't work. --- tildes/static/robots.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tildes/static/robots.txt b/tildes/static/robots.txt index 7f75347..bd2a9b7 100644 --- a/tildes/static/robots.txt +++ b/tildes/static/robots.txt @@ -43,6 +43,10 @@ Disallow: / User-agent: SemrushBot Disallow: / +# https://www.semrush.com/bot/ +User-agent: SemrushBot-BA +Disallow: / + # https://www.zoominfo.com/ User-agent: ZoominfoBot Disallow: /