The essentials
- Definition: Screaming Frog SEO Spider is desktop software (Windows, macOS, Linux) that explores a site like a search engine and lists, for each URL, the elements that matter for SEO: response code, title, description, H1, canonical, indexability, links, images, speed.
- Price: free up to 500 URLs per crawl; licence at $279 per year (checked on 16/10/2026) for unlimited crawling, advanced configurations and API connections (GA4, Search Console, PageSpeed).
- How to use it: the tool for technical audits, checks before and after a migration, and monitoring internal linking.
- Limits: a skill to learn, no continuous monitoring, no ranking or backlink data.
Screaming Frog SEO Spider is the crawler most widely used by SEO professionals. Installed on a computer, it browses a site by following links, like Googlebot, and produces a table of all URLs with their technical and editorial characteristics. It is the core tool for the full SEO audit and the essential check before and after a redesign. This guide describes its features, its free version, the ten analyses to run, the settings and the limitations.
What the crawler collects
| Family | Data per URL | What to do with it |
|---|---|---|
| Answer | HTTP code (200, 301, 404, 500), redirect chain, response time | Broken links, redirect chains, slow pages; see HTTP response codes |
| Indexability | Indexable or not, and why (noindex, canonical to another URL, robots.txt, redirect) | Important pages that cannot be indexed, useless pages that can |
| Tags | Title, meta description, H1, H2 (presence, length, duplicates) | Missing, overly long or duplicate titles |
| Example | Word count, near-duplicate or duplicate content, readability | Thin pages, duplicates; see duplicate content |
| : these aren't outbound links as far as GA4 is concerned. Create triggers in | Inbound and outbound links per page, anchors, click depth, orphan pages (with sitemap or GA4) | Internal linking; see internal linking |
| Images | File size, missing alt text, dimensions | Heavy images, accessibility |
| Structured data | Schema.org types detected, validation errors | Missing or invalid markup |
| Speed (with the PageSpeed API) | Lab and field Core Web Vitals per URL | Pages to optimise; see the Core Web Vitals |
| JavaScript rendering | Differences between raw HTML and rendered HTML | Content or links invisible without JavaScript |
Free version and licence
| Free | Licence ($279 / year) | |
|---|---|---|
| URLs per crawl | 500 | Unlimited (depending on the computer's memory) |
| Basic features | All the analyses in the table above | Same |
| Configuration | Limited | JavaScript rendering, custom extraction, custom search, scheduled crawling, URL exclusion and inclusion |
| API connections | No | GA4, Search Console, PageSpeed Insights, Ahrefs, Majestic, Moz, AI (OpenAI, Gemini, Anthropic) to analyse content |
| Saving and comparing crawls | No | Yes |
For a brochure site with fewer than 500 pages, the free version is enough for a one-off audit. The licence is worthwhile as soon as you audit regularly, want to cross-reference with Search Console and GA4, or are monitoring a migration.
The ten analyses to run on any site
- Response codes: Response Codes tab, 4xx and 5xx filters; fix or redirect.
- Redirects: chains and loops (Redirect Chains report); a single redirect to the final URL.
- Indexability: important "Non-Indexable" pages, and indexable pages with no value (parameters, archives).
- Titles and descriptions: missing, duplicate or too long; filters in the Page Titles and Meta Description tabs.
- H1: missing, multiple, or identical to the title everywhere.
- Canonicals: missing, pointing to a non-indexable URL, or inconsistent.
- Click depth: strategic pages more than 3 clicks deep; Crawl Depth tab.
- Orphan pages: URLs from the sitemap or GA4 missing from the crawl (Crawl Analysis mode with the sitemap connected).
- Images: missing alt text and file size over 200 KB; see image indexing.
- Thin and near-duplicate content: pages with fewer than 300 words, near duplicates (Near Duplicates).
Export each tab as CSV and group the fixes by template: fixing a template resolves dozens of URLs at once.
Useful settings
- Spider mode to explore from the home page; List mode to check a list of URLs (redirect map, sitemap).
- JavaScript rendering (Configuration, Spider, Rendering) for sites built with React, Vue or Angular; slower, but essential.
- Crawl speed reduced (2 to 5 URLs per second) so as not to overload small hosting.
- Googlebot user-agent to see what Google sees, while respecting robots.txt.
- Search Console and GA4 connection to add clicks, impressions and sessions to each URL: pages with no traffic and no impressions are candidates for merging or removal.
- Custom extraction (XPath, regex) to check that an element is present on every page: consent tag, price, author.
- Crawl comparison before and after a change or a migration.
Limitations and alternatives
| Limit | Alternative or complement |
|---|---|
| Desktop tool, one-off use, no continuous monitoring | Online crawlers with monitoring (Sitebulb Cloud, Oncrawl, Lumar) or the site health module of a management platform |
| Large sites (over 500,000 URLs) limited by memory | Screaming Frog's database storage mode, or cloud crawlers |
| No rankings, keywords or backlinks | Search Console, Monitorank, Ahrefs, Majestic; see Majestic SEO |
| A skill to learn | Sitebulb (more guided, from £13 per month) for beginners |
| Only sees what is linked | Connect the sitemap and GA4 to find orphan pages |
How GreenRed helps
Rather than juggling several tools, GreenRed's Site health module brings these metrics together in a single dashboard, compares them over time and tells you which actions come first. You can try it free, with no card, from the Pricing.
Frequently asked questions
Is Screaming Frog free?
The free version crawls up to 500 URLs per crawl with all the basic analyses, which is enough for a brochure site. The licence, at $279 per year, removes the limit and adds JavaScript rendering, API connections (Search Console, GA4, PageSpeed), custom extraction, and saving and comparing crawls.
Does Screaming Frog replace Semrush or Ahrefs?
No: it covers technical auditing and internal linking (which the suites do less well), but provides no rankings, keywords or backlinks. It is used alongside Search Console (free) for rankings and a backlink tool for link building.
Should you enable JavaScript rendering?
Yes, if the site is built with a JavaScript framework (React, Vue, Angular, Next.js with client-side rendering) or loads content and links with JavaScript. The crawl is slower, but it shows what Google actually indexes. For a standard WordPress or static site, an HTML crawl is enough.
How often should you crawl your site?
After every major release (redesign, migration, new template) and at least once a quarter for an active site. Sites that publish a lot or change often benefit from adding continuous monitoring with a cloud crawler or a site health module.