Key Takeaways
- Definition: Screaming Frog SEO Spider is a desktop application (Windows, macOS, Linux) that crawls a website like a search engine and lists, for each URL, the elements that matter for SEO: response code, title, description, H1, canonical tag, indexability, links, images, and page speed.
- Price: Free for up to 500 URLs per crawl; a license costs €259 (excluding tax) per year for unlimited crawls, advanced configurations, and API connections (GA4, Search Console, PageSpeed).
- Usage: a tool for technical audits, pre- and post-migration checks, and monitoring of the internal network.
- Limitations: a skill to learn, no ongoing monitoring, no position data or backlinks.
Screaming Frog SEO Spider is the crawler most commonly used by SEO professionals. Installed on a computer, it crawls a website by following links—just like Googlebot—and generates a table listing all the URLs along with their technical and editorial characteristics. It is the essential tool for The Comprehensive SEO Audit and the essential checks to perform before and after a redesign. This guide describes its features, its free version, the ten analyses to perform, the settings, and the limitations.
What the crawler collects
| Family | Data by URL | What We Do With It |
|---|---|---|
| Answer | HTTP status codes (200, 301, 404, 500), redirect chain, response time | Broken links, chained redirects, slow-loading pages; see HTTP response codes |
| Indexability | Indexable or not, reason (noindex, canonical tag pointing to another URL, robots.txt, redirect) | Important pages that cannot be indexed, useless pages that can be indexed |
| Tags | Title, meta description, H1, H2 (presence, length, duplicates) | Missing, excessively long, or duplicate titles |
| Contents | Word count, similar or duplicate content, readability | Thin pages, duplicate pages; see duplicate content |
| Links | Inbound and outbound links per page, anchor text, click depth, orphaned pages (using a sitemap or GA4) | Internal linking; see internal linking |
| Images | Weight, missing alt text, dimensions | Large Images, Accessibility |
| Structured Data | Detected Schema.org types, validation errors | Missing or invalid markup |
| Speed (using PageSpeed API) | Core Web Vitals: Lab and Field Data by URL | Pages to optimize; see Core Web Vitals |
| JavaScript Rendering | Differences Between Raw HTML and Rendered HTML | Content or links that are invisible without JavaScript |
Free Version and License
| Free | Bachelor's Degree (€259 plus tax per year) | |
|---|---|---|
| URLs per crawl | 500 | Unlimited (depending on the computer's memory) |
| Basic Functions | All the analyses in the table above | Same as above |
| Configuration | Limited | JavaScript rendering, custom extraction, custom search, scheduled crawling, URL exclusion and inclusion |
| API Connections | No | GA4, Search Console, PageSpeed Insights, Ahrefs, Majestic, Moz, AI (OpenAI, Gemini, Anthropic) for analyzing content |
| Backing Up and Comparing Crawls | No | Yes |
For a showcase website with fewer than 500 pages, the free version is sufficient for a one-time audit. The paid license is worth it if you conduct audits regularly, want to cross-reference data with Search Console and GA4, or are overseeing a migration.
The Ten Tests to Be Conducted at Every Site
- Response codes: "Response Codes" tab, 4xx and 5xx filters; correct or redirect.
- Redirects: chains and loops (Redirect Chains report); a single redirect to the final URL.
- Indexability: Important "non-indexable" pages, and indexable pages with no value (parameters, archives).
- Titles and descriptions: missing, duplicated, or too long; filters on the “Page Titles” and “Meta Description” tabs.
- H1: missing, multiple, identical to the title everywhere.
- Canonicals: missing, pointing to a non-indexable URL, or inconsistent.
- Click depth: Strategic pages more than 3 clicks away; Crawl Depth tab.
- Orphan pages: Sitemap or GA4 URLs are missing from the crawl (Crawl Analysis mode with sitemap connected).
- Images: Missing alt text and file sizes larger than 200 KB; see image indexing.
- Concise and relevant content: Pages with fewer than 300 words, near duplicates.
Export each tab as a CSV file and group the corrections by template: a single template correction fixes dozens of URLs at once.
Useful Settings
- Spider Mode to explore from the home page; List mode to check a list of URLs (redirect table, sitemap).
- JavaScript Rendering (Configuration, Spider, Rendering) for React, Vue, or Angular sites; slower, but essential.
- Crawl Speed reduced (2 to 5 URLs per second) to avoid overloading a small hosting service.
- User-agent Googlebot to see what Google sees, while respecting the robots.txt file.
- Search Console and GA4 Integration To add clicks, impressions, and sessions to each URL: Pages with no traffic or impressions are candidates for merging or deletion.
- Custom Extraction (XPath, regex) to check for the presence of an element on all pages: consent tag, price, author.
- Comparison of Crawls before and after a change or migration.
Limitations and Alternatives
| Limit | Alternative or Supplement |
|---|---|
| Desktop tool, one-time use, no continuous monitoring | Online crawlers with tracking (Sitebulb Cloud, Oncrawl, Lumar) or a site health module from a management platform |
| Large websites (more than 500,000 URLs) limited by memory | Screaming Frog's database mode, or cloud crawlers |
| No rankings, keywords, or backlinks | Search Console, Monitorank, Ahrefs, Majestic; see Majestic SEO |
| Skill to be acquired | Sitebulb (more guided, starting at £13 per month) for beginners |
| See only what is related | Connect the sitemap and GA4 for orphaned pages |
How GreenRed Can Help You
Instead of juggling multiple tools, the Health module on the GreenRed website brings these metrics together in a single dashboard, compares them over time, and shows you which actions to prioritize. You can try it for free—no credit card required—on the page Rates.
Frequently Asked Questions
Is Screaming Frog free?
The free version crawls up to 500 URLs per crawl and includes all basic analyses, which is sufficient for a showcase site. The license, priced at €259 per year (excluding tax), removes the limit and adds JavaScript rendering, API connections (Search Console, GA4, PageSpeed), custom data extraction, backup, and crawl comparison.
Does Screaming Frog replace Semrush or Ahrefs?
No: It covers technical audits and link mapping (which other suites don't do as well), but it doesn't provide rankings, keywords, or backlinks. It works in conjunction with Search Console (which is free) for rankings and with a backlink tool for link building.
Should JavaScript rendering be enabled?
Yes, if the site is built using a JavaScript framework (React, Vue, Angular, Next.js with client-side rendering) or loads content and links via JavaScript. The crawl is slower, but it shows what Google actually indexes. For a WordPress site or a standard static site, an HTML crawl is sufficient.
How often should you crawl your website?
After every major update (redesign, migration, new template) and at least once a quarter for a live site. Sites that publish frequently or undergo frequent changes would benefit from adding continuous monitoring using a cloud crawler or a site health module.