Browse Source

robots.txt: Block /login from all scrapers

Various scrapers hit this a lot because it's on all pages with a
differing query variable (for the "from url" to redirect back to after
logging in).
merge-requests/82/merge
Deimos 5 years ago
parent
commit
7c33144995
  1. 6
      tildes/static/robots.txt

6
tildes/static/robots.txt

@ -4,6 +4,12 @@
# Scrapers from SEO/marketing-type services will be blocked. Tildes data is not a # Scrapers from SEO/marketing-type services will be blocked. Tildes data is not a
# resource to be mined and sold. # resource to be mined and sold.
# urls blocked for all scrapers
User-agent: *
Disallow: /login
# Specific, fully-blocked scrapers below
# https://ahrefs.com/robot # https://ahrefs.com/robot
User-agent: AhrefsBot User-agent: AhrefsBot
Disallow: / Disallow: /

Loading…
Cancel
Save