Mirror
/
tildes
mirror of https://gitlab.com/tildes/tildes.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Browse Source

Block Cloudfind scraper

merge-requests/85/head
Deimos 5 years ago
parent
4657521373
commit
ebb6c911d2
1 changed files with 4 additions and 0 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 4
      tildes/static/robots.txt

4
tildes/static/robots.txt
View File

@ -22,6 +22,10 @@ Disallow: /
User-agent: Buck User-agent: Buck
Disallow: / Disallow: /
# https://cloudfindhq.com/
User-agent: Cloudfind
Disallow: /
# https://moz.com/help/moz-procedures/crawlers/dotbot # https://moz.com/help/moz-procedures/crawlers/dotbot
User-agent: dotbot User-agent: dotbot
Disallow: / Disallow: /

Write Preview
Loading…
Cancel
Save