Comments63Characters 1207Views4min1sRead
Are you noticing sudden server crashes, slow loading times, or the dreaded “508 Resource Limit Exceeded” error on your WordPress site? You are not alone.
In 2026, a new threat has emerged for independent website owners: aggressive AI web crawlers.
Companies like OpenAI, Anthropic, Google, and dozens of AI startups are constantly scraping the web to train their large language models (LLMs). Unlike traditional search engine bots—such as Googlebot—that crawl your site gently, AI bots often hammer your server with thousands of simultaneous requests. This can quickly drain your CPU and RAM, causing your WordPress site to go offline.
In this step-by-step guide, you will learn exactly how to identify, track, and block aggressive AI bots to save your server resources and boost your site speed.
Note: The official HTTP specification defines status code 508 as “Loop Detected” for WebDAV systems. However, nearly all cPanel & CloudLinux shared hosting providers repurpose 508 to signal Resource Limit Exceeded—meaning your site has hit its CPU, RAM, or entry process cap. This is the error we cover in this guide.
Step 1: Identify if AI Bots Are Crashing Your WordPress Site
Before changing any settings, you need to verify if AI crawlers are the real reason behind your high server load.
Check Your Server Access Logs
Log in to your hosting control panel (cPanel, hPanel, or RunCloud) and open your Access Logs. Look for the following AI-related User-Agents:
Important:
Google-Extended is not a User-Agent you will see in raw access logs. It is a robots.txt token used to tell Google not to use your content for AI training. Google’s crawler still appears as Googlebot. If you see Googlebot making an unusually high number of rapid requests, it could be scraping for Gemini-related training purposes. We will cover how to address that in the next step.If you see any of the AI User-Agents above repeating hundreds of times per minute, your site is likely being aggressively scraped.
Step 2: Block AI Crawlers Using robots.txt (The Easiest Way)
The most ethical way to tell AI companies to stop using your content and bandwidth is by updating your
robots.txt file.Add the following code:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
Why this matters: Well-known companies like OpenAI, Anthropic, and Google publicly state that they respect
robots.txt directives. Adding Google-Extended specifically tells Google not to use your site content for training its Gemini model.However, many rogue AI scrapers completely ignore
robots.txt. That is why you need the next step.Step 3: Block Rogue AI Crawlers with .htaccess (Server-Level Block)
If AI bots ignore your
robots.txt and keep spamming your server, you can block them at the server level using your .htaccess file. This method works for Apache and LiteSpeed servers.Before you edit, please back up your
.htaccess file. A simple mistake can make your site inaccessible.- Log in to your hosting File Manager and locate the
.htaccessfile in yourpublic_htmlfolder. - Open the file for editing.
- Critical placement rule: Paste the custom rule before the
# BEGIN WordPressline at the top of the file, or insert it directly afterRewriteEngine Oninside the existing WordPress rewrite block. - Do not add another
RewriteEngine Online if WordPress already includes one. - Add the following code:
This tells your server to immediately return a 403 Forbidden error whenever these specific AI bots try to access your site, saving your PHP memory and CPU power.
For Nginx Servers
If your host uses Nginx instead of Apache,
.htaccess will not work. In that case, add the following snippet inside your site’s server block configuration, then reload Nginx:Step 4: The Ultimate Solution – Deploy Cloudflare WAF Rules
The absolute best way to stop AI bots from crashing your WordPress site is to block them before they even reach your server. You can achieve this easily with Cloudflare’s free Web Application Firewall (WAF).
Since Cloudflare sits in front of your hosting provider, it handles bot traffic on its global network. This means your actual web hosting uses 0% resources to block them.
The one-click method (free on all plans)
As of 2026, Cloudflare provides a built-in toggle for this exact problem, available even on the Free plan:
- Log in to your Cloudflare Dashboard.
- Go to Security → Bots.
- Turn on the switch labeled “Block AI Scrapers and Crawlers”.
- Done. Cloudflare will instantly stop known AI harvesters at the edge.
Custom WAF rule (optional, more control)
If you prefer to manually define which bots to challenge or block:
- Go to Security → WAF → Custom Rules and click Create Rule.
- Set the condition: User Agent → contains →
GPTBot. - Click Or to add more User-Agent strings like
ClaudeBot,PerplexityBot,Bytespider. - Set the action to Block or Managed Challenge.
- Deploy.
Frequently Asked Questions
Will blocking AI bots hurt my regular Google SEO?
No. The rules in this guide only target AI training crawlers like GPTBot and ClaudeBot. They do not affect Googlebot, Bingbot, or standard search engine crawlers, so your organic search rankings should remain unchanged.
Can rogue AI scrapers ignore robots.txt?
Yes.
robots.txt is a voluntary guideline, not a security enforcement tool. Reputable companies like OpenAI and Google generally follow it, but unethical scrapers may forge their user agent, ignore the file entirely, and continue hitting your server. This is why server-level and Cloudflare blocks are more reliable.Do I need Cloudflare Pro to block AI scrapers?
No. As of 2026, the free Cloudflare plan includes a one-click toggle to block AI scrapers. You can also create up to 5 custom WAF rules for free if you want more granular control.
I blocked AI bots but still get 508 errors. What else can I do?
If 508 errors continue during normal human traffic, your hosting plan’s resource limits are likely too low. Other common causes include bloated plugins, unoptimized databases, brute-force attacks on
xmlrpc.php, or insufficient server resources. In that case, blocking bots alone will not fix the problem.Summary: Upgrade Your Hosting If 508 Errors Persist
Blocking AI bots will significantly reduce your server load. However, if your WordPress site continues to show “508 Resource Limit Exceeded” errors during normal traffic spikes, it is a clear sign that your current shared hosting plan is no longer enough.
For resource-heavy WordPress sites, blogs, and page builder-powered pages, we highly recommend upgrading to a managed cloud hosting plan with dedicated CPU and RAM. Providers like Hostinger Cloud Hosting or Cloudways VPS ensure your website stays fast and online, no matter how many bots or real visitors hit your pages.
Did this guide help you fix your WordPress 508 error? Let us know in the comments which AI bot was causing the most trouble for your server!
How to Set Up Automatic Excerpt Generation in WordPress
WordPress by default displays the first portion of your post content as an excerpt on the homepage...
How to Fix WordPress Not Sending Password Reset Emails
If your WordPress site fails to send password reset emails, you are facing a critical functionality ...