What Is Screaming Frog SEO Spider?
Also known as: Screaming Frog, SEO Spider
Screaming Frog SEO Spider is a desktop website crawler that scans sites for technical SEO issues — broken links, redirects, metadata, structured data, duplicate content and more.
Last reviewed: 18 August 2026 · Reviewed by iNet Ventures Technical SEO Team
In plain English
You install it on your computer, type in a website address, and it crawls every page it can find — just like Googlebot does. As it goes, it records every URL, every link, every title tag, every redirect, and every error. When it finishes, you get a spreadsheet-like view of everything it found, which you use to identify and fix technical problems before they hurt your search performance.
Technical definition
Screaming Frog SEO Spider is a desktop application (available for Windows, macOS and Linux) that crawls websites from a given starting URL, following internal links to discover pages. It renders pages using a headless Chromium engine, extracts on-page elements and technical signals, and presents them in a filterable interface with export capabilities. The free version crawls up to 500 URLs; the paid licence removes that limit and adds features like JavaScript rendering, scheduling, search console integration, and API connections.
Quick example
Point the spider at a new client domain, let it crawl, then open the Response Codes tab filtered to 4xx and 5xx. You immediately see every broken internal link, every dead redirect target, and every server error — the foundational technical issues that need fixing before any content or link building work begins.
Quick facts
| Meaning | A desktop website crawler for technical SEO auditing |
|---|---|
| Main category | Technical SEO tools |
| Used by | SEO agencies, technical SEOs, in-house teams, web developers |
| Difficulty | Intermediate |
| Importance | High |
| Common examples | Technical site audits, broken link checking, redirect mapping, metadata extraction, duplicate content analysis |
| Still relevant? | Yes. It is the most widely used standalone site crawler for technical SEO audits. The free tier (500 URLs) is sufficient for small sites, and the paid licence is standard equipment for agency audit workflows. |
Why Screaming Frog SEO Spider matters
Screaming Frog matters because technical issues are invisible without crawling. You cannot fix what you cannot see, and most CMS dashboards do not surface broken links, redirect chains, canonical conflicts, or orphaned pages. A dedicated crawler is the only reliable way to find them.
It finds issues before Google does
If Googlebot encounters broken links, redirect loops, or noindex tags during a crawl, it wastes crawl budget and can deindex pages. Running the spider regularly catches these issues before they affect rankings.
It produces actionable export data
Every tab can be exported to CSV or Excel, which means audit findings can be sorted, filtered, prioritised and handed to developers as a work list. This is how technical SEO recommendations get implemented rather than ignored.
It handles JavaScript-rendered content
Modern sites increasingly rely on JavaScript to render content. The spider's rendering mode uses a headless Chromium browser to execute JavaScript, so it sees what Googlebot sees — not just the raw HTML source.
It scales to large sites
With configurable crawl limits, URI exclusion patterns, and crawl depth settings, the spider can handle sites with hundreds of thousands of URLs without crashing or producing unmanageable output.
It integrates with other tools
The spider can connect to Google Search Console, Google Analytics, and PageSpeed Insights API, enriching crawl data with real performance metrics. This turns a static audit into a prioritised action plan.
How it works
The spider works by fetching URLs, parsing their HTML, extracting links, and following those links — the same basic process as a search engine crawler.
- 1
You configure the crawl
Set the starting URL, crawl mode (website, list, or upload), user-agent, crawl depth, speed limits, and exclusion patterns. Configuration matters — crawling a large site with default settings can take hours or miss important sections.
- 2
The spider fetches each URL
It sends an HTTP request to each URL, records the response code, headers, and HTML content. If rendering mode is enabled, it also executes JavaScript and captures the rendered DOM.
- 3
On-page elements are extracted
From each page, the spider extracts title tags, meta descriptions, H1-H6 headings, canonical tags, meta robots directives, hreflang attributes, structured data, word count, and internal/external links.
- 4
Links are discovered and queued
Every internal link found is added to the crawl queue (unless excluded by configuration). External links are recorded but not followed. This is how the spider discovers pages — by following the site's own internal linking structure.
- 5
Data is presented in tabs and exportable reports
When the crawl completes, data is organised into tabs — Response Codes, Redirects, Titles, Meta Descriptions, Images, Directives, Structured Data, and more. Each tab supports filtering and export.
Types and variations
The spider is described by its crawl modes and key features.
Spider Mode
The default mode. Enter a URL and the spider crawls the site by following internal links. Best for general site audits.
List Mode
Upload a list of URLs (from a sitemap, log file, or export) and the spider crawls only those pages. Useful for checking specific pages or validating redirect maps.
Rendering Mode
Uses a headless Chromium browser to execute JavaScript and capture the rendered DOM. Essential for auditing JavaScript-heavy sites, SPAs, and content loaded via AJAX.
Crawl Configuration
Settings that control what the spider crawls — exclude patterns, include patterns, crawl depth, max URI limit, and speed. Proper configuration is critical for large sites to avoid excessive crawl times.
Search Console Integration
Connects to Google Search Console to pull impression, click, and position data for crawled URLs. Adds a performance dimension to technical findings.
PageSpeed Insights Integration
Connects to the PageSpeed Insights API to fetch Core Web Vitals and performance scores for crawled URLs. Identifies pages that need speed optimisation.
Scheduled Crawls
Available in paid licences. Schedule crawls to run automatically on a recurring basis, with results saved to a specified location. Useful for monitoring sites over time.
Real examples
Concrete ways the spider gets used in audit workflows.
Redirect mapping for a site migration
Before migrating a site, crawl the old domain in List Mode using the existing sitemap. Export all URLs with their status codes. After migration, crawl the new domain and compare. Every old URL should 301 redirect to its new equivalent — any that return 404s or redirect to the wrong page are migration bugs that need immediate fixing.
Finding duplicate content issues
After a crawl, open the Titles or Meta Descriptions tab and filter for duplicates. Pages with identical titles or descriptions compete with each other in search results and dilute relevance signals. Each duplicate group needs a unique title or a canonical tag pointing to the preferred version.
Crawling without checking robots.txt
Running a crawl with "respect robots.txt" disabled on a client site. The spider follows links that Googlebot would not, producing findings that are irrelevant to actual search performance. Always respect robots.txt unless you have a specific reason not to.
Crawling a 500,000-URL site with default settings
Accepting the default 500-URL limit on a large site and reporting "no issues found." The spider only crawled a fraction of the site. Configure crawl limits, exclusion patterns, and speed settings appropriate to the site size, or use List Mode with a sitemap to ensure comprehensive coverage.
Best practices
The spider is only as useful as the configuration and analysis applied to its output.
What to do
- Always configure exclusion patterns for parameter URLs, faceted navigation, and staging environments
- Use rendering mode for JavaScript-heavy sites — the default non-rendered crawl will miss content
- Export findings to CSV and sort by priority — 5xx errors before 4xx, 4xx before redirects, redirects before metadata
- Cross-reference crawl data with Search Console and GA4 to prioritise fixes by traffic impact
- Save crawl configurations per client so audits are repeatable and comparable over time
- Use List Mode with a sitemap URL to verify that all submitted URLs are crawlable and return 200
- Check the Internal tab for orphaned pages — URLs that exist but have no internal links pointing to them
What to avoid
- Running crawls with default settings on large or complex sites
- Disabling robots.txt respect without a specific technical reason
- Reporting every minor metadata issue as critical — prioritise by page value and traffic
- Forgetting to check the Redirects tab for chains and loops — these waste crawl budget and pass reduced link equity
- Ignoring the Structured Data tab — invalid markup can prevent rich results
- Crawling production sites at maximum speed — throttle the crawler to avoid impacting server performance
Common mistakes
Not configuring crawl limits for large sites
The default 500-URL limit on the free version will stop a crawl partway through, giving an incomplete picture. On paid licences, not setting a max URI limit on a site with millions of URLs can result in extremely long crawl times and huge memory usage.
Assuming non-rendered crawl shows everything
Without rendering mode, the spider sees only the raw HTML source. Content injected by JavaScript — including single-page application content, lazy-loaded images, and AJAX-rendered text — will be missing. Always enable rendering for JS-heavy sites.
Ignoring the Crawl Analysis report
The Crawl Analysis feature runs additional checks on crawled data, including duplicate content, thin pages, and missing alt text. Skipping it means missing findings that are not visible in the main tabs.
Not saving crawl configurations
Each client site has different crawl requirements — exclusion patterns, speed limits, rendering settings. If you do not save these as named configurations, the next audit will use different settings and the results will not be comparable.
Overwhelming developers with raw exports
Exporting 50,000 rows of crawl data and sending it to a development team is not actionable. Filter, sort, and prioritise findings — group by issue type, highlight high-traffic affected pages, and provide specific fix recommendations.
Forgetting to check external links
The External tab shows all outbound links and their status codes. Broken external links (4xx and 5xx) are a poor user experience and waste link equity. Check and fix them as part of any audit.
Why a dedicated crawler still matters in the age of cloud tools
Cloud-based SEO platforms like Semrush and Ahrefs include site audit tools, but they run on their servers with limited crawl depths and configuration options. Screaming Frog runs locally, which means full control over crawl speed, rendering, exclusions, and memory allocation. For technical SEOs auditing large or complex sites, that control is essential. The trade-off is that it requires a desktop machine with sufficient RAM, and crawls of very large sites can take hours. The two approaches are complementary — cloud audits for ongoing monitoring, Screaming Frog for deep-dive investigations.
- Local crawling gives full control over configuration — cloud tools cannot match this flexibility
- JavaScript rendering in Screaming Frog uses real Chromium, matching Googlebot's rendering pipeline
- API integrations with Search Console and PageSpeed Insights add performance context to technical findings
- For agencies, the one-time licence fee is often more cost-effective than ongoing cloud audit subscriptions
How iNet Ventures approaches screaming frog seo spider
We run Screaming Frog on every new client engagement as part of our technical SEO baseline. It is the fastest way to identify the foundational issues — broken links, redirect chains, missing metadata, canonical conflicts — that would undermine any link building or content work. We export findings, prioritise by traffic impact, and deliver them as a developer-ready fix list.
Initial crawl
We configure the spider with client-specific exclusion patterns, enable rendering for JS-heavy sites, and run a full crawl. For large sites, we use List Mode with the sitemap to ensure comprehensive coverage.
Finding prioritisation
We export crawl data, cross-reference with Search Console traffic data, and prioritise fixes by impact. A broken page with 1,000 monthly visits is fixed before a missing meta description on a page with 10 visits.
Ongoing monitoring
For retainer clients, we schedule recurring crawls and compare results over time. New issues introduced by content updates or development changes are caught before they affect rankings.
Continue learning
Best SEO Link Building Tools
How Screaming Frog fits into a broader SEO tool stack.
Technical SEO Audit
Professional site audits using Screaming Frog and Search Console data.
Link Building Plans
Monthly link building with technical foundations in place.
Best Blogger Outreach Services
Compare top outreach providers and their approaches.
Free Backlink Analyzer
Check a backlink profile without a paid subscription.
Frequently asked questions
Is Screaming Frog free?
The free version crawls up to 500 URLs, which is sufficient for small sites and evaluation. The paid licence removes the URL limit and adds features like JavaScript rendering, scheduled crawls, Search Console integration, and API connections. The licence is a one-time annual payment, not a monthly subscription.
How does Screaming Frog compare to cloud-based audit tools?
Screaming Frog runs locally, giving full control over crawl configuration, speed, and rendering. Cloud tools like Semrush Site Audit run on remote servers with limited configuration options but offer automatic scheduling and easier collaboration. Most technical SEOs use both — Screaming Frog for deep audits and cloud tools for ongoing monitoring.
Does Screaming Frog crawl JavaScript?
Yes, with rendering mode enabled. The spider uses a headless Chromium browser to execute JavaScript and capture the rendered DOM, showing content that is injected by JS frameworks, AJAX, or lazy loading. This is essential for auditing modern single-page applications and JavaScript-heavy sites.
How long does a Screaming Frog crawl take?
It depends on the site size, crawl speed settings, and whether rendering is enabled. A 500-URL site takes minutes. A 100,000-URL site with rendering enabled can take several hours. Configure speed limits and exclusion patterns to optimise crawl time for large sites.
Can Screaming Frog find broken links?
Yes. The Response Codes tab shows every URL the spider encountered and its HTTP status. Filter to 4xx (client errors) and 5xx (server errors) to find broken internal and external links. The Inlinks tab for any 404 URL shows which pages link to it, so you know exactly where to fix the broken link.
Does Screaming Frog work on Mac?
Yes. Screaming Frog SEO Spider is available for Windows, macOS, and Linux. The macOS version requires macOS 10.15 or later. All versions have the same features and the same free 500-URL limit.
Written from daily hands-on use of Screaming Frog SEO Spider across client site audits ranging from 50-page brochure sites to 500,000-URL e-commerce platforms. Configuration recommendations are based on real-world crawl experience.
- Written by:
- iNet Ventures Editorial Team
- Reviewed by:
- iNet Ventures Technical SEO Team
- Published:
- 18 August 2026
- Last reviewed:
- 18 August 2026
Sources and further reading
Spotted something out of date or incorrect? Tell us and we will review and correct it.
Need links that actually hold up?
White-label link building and digital PR, priced per placement. Every link comes with a live URL and the context behind it.