The essentials
- Definition: crawl budget is the number of URLs a search engine crawls on a site in a given time; it combines your server's capacity with the engine's interest in the site.
- Who it affects: sites with more than 10,000 pages, online stores with filters, and sites that publish heavily; a 200-page site has no crawl budget problem.
- The waste: parameter URLs, endless pagination, redirected or 404 pages, duplicate content, indexable pages with no value.
- Levers: server speed, robots.txt, canonicals, noindex, a clean sitemap, internal links to the useful pages, and removing worthless pages.
The aim of crawl budget is the number of URLs a search engine crawls on a site over a given period. Google describes it as the combination of two things: the capacity limit, meaning what your server can take without slowing down, and crawl demand, meaning how interested Google is in your pages according to their popularity and freshness. When a site has more pages than Google crawls, new or updated pages wait days or weeks before being seen.
Which sites are affected
| Type of site | Affected? | Why |
|---|---|---|
| A brochure site of 20 to 500 pages | No | Google crawls all of it within days |
| A blog or publication of 500 to 10,000 pages | Rarely | Only if pagination and archives create tens of thousands of URLs |
| An online store with filters and sorting | Yes | Every filter combination creates a URL; millions of possible URLs |
| A site of more than 10,000 pages, or publishing daily | Yes | The volume exceeds what Google crawls per day |
| A slow site, or one with frequent errors | Yes, whatever its size | Google reduces crawling so as not to overload the server |
For a small business with a few hundred pages, the question isn't crawl budget but indexing: a crawled page may not be indexed if Google judges it unhelpful. Both are read in the same Search Console report; see indexing in Search Console.
What crawl budget depends on
- Server speed and availability: a short response time and no 5xx errors raise capacity; a slow server lowers it immediately.
- Popularity: pages receiving external and internal links are crawled more often; see PageRank.
- Freshness: a site that publishes or updates regularly is visited more often.
- Quality: duplicated, empty or low-value pages reduce crawl demand for the whole site.
- Structure: pages reachable in three clicks from the home page are crawled before deep pages.
What wastes the budget
- Parameter URLs: sorting, filters, session IDs, internal UTMs; thousands of URLs for the same content.
- Endless pagination and archives: list pages, tags, calendars.
- Redirect chains and 404s: every hop and every error consumes a crawl; see the 301 redirect.
- Duplicate content: http and https, with and without www, with and without a trailing slash, printable versions; see the canonical tag.
- Indexable pages with no value: internal search results, baskets, accounts, test pages.
- Heavy resources: unnecessary scripts and images that slow rendering.
Reading crawl budget in Search Console
The Crawl stats report (Settings, then Crawl stats) shows the number of requests per day, the volume downloaded, the average response time, and the breakdown by response code, by file type and by purpose (discovery or refresh). A response time rising above 500ms, a share of 404s or redirects above 10%, or a majority of crawls landing on parameter URLs all signal a badly used budget. Your server log files give the detail URL by URL; a crawler such as Screaming Frog reproduces Google's path and lists the useless URLs.
Optimising crawl budget
| Lever | Action |
|---|---|
| Server | Bring the response time below 300ms, fix 5xx errors, add caching; see the Core Web Vitals |
| Robots.txt | Block parameter URLs, internal search, baskets; see the robots.txt file |
| Canonicals and noindex | One indexable version per piece of content; noindex on pages useful to visitors but not to the engine |
| Sitemap | Only canonical URLs returning 200, with a reliable lastmod; see the XML sitemap |
| Internal linking | Important pages within three clicks, links from your popular pages |
| Cleaning up | Delete or merge pages with no traffic and no links, fix 404s and redirect chains |
How GreenRed helps
Rather than juggling several tools, GreenRed's Site health module brings these metrics together in a single dashboard, compares them over time and tells you which actions come first. You can try it free, with no card, from the Pricing.
Frequently asked questions
What is crawl budget?
The number of URLs a search engine crawls on a site over a given period. Google defines it as the combination of server capacity (what it can take without slowing down) and crawl demand (the engine's interest in your pages, according to their popularity and freshness).
Does my 300-page site have a crawl budget problem?
No, unless it is very slow or frequently erroring. Google crawls a site that size in full within days. If pages aren't indexed, the cause is their quality or their internal linking, not crawl budget; the Pages report in Search Console gives the reason.
How do I know whether Google is wasting its crawling on my site?
In Search Console's Crawl stats: a response time above 500ms, more than 10% of 404s or redirects, or a majority of crawls landing on parameter URLs. Your server log files and a crawl of the site give the detail.
Which levers are the most effective?
Speeding up the server, blocking parameter URLs in robots.txt, enforcing one indexable version per piece of content (canonical, noindex), cleaning the sitemap, fixing 404s and redirect chains, and bringing important pages closer to the home page through internal linking.