#rate limit yandex (high traffic) User-agent: Yandex Crawl-delay: 60 Disallow: /wp-admin/ Disallow: /?blackhole #disallow ahrefsbot #https://ahrefs.com/robot User-agent: AhrefsBot Disallow: / #allow Google bot access to most things User-agent: Googlebot Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?blackhole #rate limit all other bots a bit User-agent: * Crawl-delay: 5 Disallow: /wp-admin/ Disallow: /?blackhole