Block moz.com dotbot from crawling the site

6 years ago · f87d84759d
1 changed files with 3 additions and 0 deletions
--- a/tildes/static/robots.txt
+++ b/tildes/static/robots.txt
@ -1,2 +1,5 @@
 User-agent: BLEXBot
 Disallow: /
+
+User-agent: dotbot
+Disallow: /