Browse Source

Block SemrushBot in nginx (it ignores robots.txt)

merge-requests/86/head
Deimos 5 years ago
parent
commit
282df2bf02
  1. 5
      salt/salt/nginx/tildes.conf.jinja2

5
salt/salt/nginx/tildes.conf.jinja2

@ -13,6 +13,11 @@ map $sent_http_content_type $expires_type_map {
}
server {
# block bots that don't obey robots.txt
if ($http_user_agent ~* (SemrushBot)) {
return 403;
}
# remove trailing slash from addresses, the $port thing is a hack for
# development in Vagrant, so the port forwarding from the host is kept
set $port '';

Loading…
Cancel
Save