Mirror
/
tildes
mirror of https://gitlab.com/tildes/tildes.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Browse Source

robots.txt: Block Majestic scraper (MJ12bot)

merge-requests/82/merge
Deimos 5 years ago
parent
673ffd26e1
commit
983bae7039
1 changed files with 5 additions and 0 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 5
      tildes/static/robots.txt

5
tildes/static/robots.txt
View File

@ -28,6 +28,11 @@ Disallow: /
User-agent: MegaIndex User-agent: MegaIndex
Disallow: / Disallow: /
# https://mj12bot.com/
# Note: claims to be powering a search engine, actually sells SEO/marketing data
User-agent: MJ12bot
Disallow: /
# https://www.semrush.com/bot/ # https://www.semrush.com/bot/
User-agent: SemrushBot User-agent: SemrushBot
Disallow: / Disallow: /
Write Preview
Loading…
Cancel
Save