iNet Ventures logoiNet Ventures
Crawlability, speed and structure

Technical SEO Glossary

The technical foundations of search visibility — crawl budget, indexation, canonical tags, structured data, Core Web Vitals, redirects and site architecture.

About Technical SEO

Technical SEO makes everything else possible. If a page cannot be crawled, rendered or indexed correctly, no amount of content or links will help it rank. These terms cover the infrastructure layer of search.

Key things to know

  • Crawlable and indexable comes before optimised — always fix this order first
  • Canonical tags are a hint to Google, not a directive it must obey
  • Core Web Vitals are a confirmed but relatively lightweight ranking factor
  • Structured data helps understanding and rich results, but does not boost rankings directly
In-depth guides

Key terms explained in full

Intermediate

Canonical Tag

A canonical tag is an HTML link element, rel="canonical", that indicates which URL out of a group of duplicate or near-duplicate pages should be treated as the primary, "master" version for indexing.

Read the full guide
Intermediate

Core Web Vitals

Core Web Vitals are three specific, standardised metrics — Largest Contentful Paint (LCP), Interaction to Next Paint (INP), and Cumulative Layout Shift (CLS) — that Google uses to measure real-world user experience on a page.

Read the full guide
Intermediate

Cumulative Layout Shift

Cumulative Layout Shift (CLS) measures visual stability by tracking how much page elements move unexpectedly during load.

Read the full guide
Advanced

Interaction to Next Paint

Interaction to Next Paint (INP) is a Core Web Vitals metric that measures how quickly a page responds to user interactions like clicks, taps, and key presses.

Read the full guide
Beginner

Internal Linking

Internal linking is the practice of connecting pages within the same website via hyperlinks, helping both users and search engines navigate and understand the relative structure and importance of a site's content.

Read the full guide
Intermediate

Largest Contentful Paint

Largest Contentful Paint (LCP) measures how long it takes for the largest visible content element to load within the viewport.

Read the full guide
Beginner

Meta Description

A meta description is an HTML meta tag in a page's head section that provides a short summary of the page's content, which search engines may use as the descriptive snippet shown beneath a result's title in search listings.

Read the full guide
Intermediate

Mobile-First Indexing

Mobile-first indexing means Google predominantly uses the mobile version of a site's content, crawled using a smartphone user agent, for indexing and ranking purposes, rather than the desktop version.

Read the full guide
Intermediate

Schema Markup

Schema markup is structured data added to a page's code, using the shared Schema.org vocabulary, that explicitly describes the page's content to search engines in a standardised, machine-readable format.

Read the full guide

All Technical SEO terms

Search or browse every term in this category. Click any term to read the full definition, or open a full guide where one exists.

Showing 52 of 52 terms

Article Schema

Article schema is structured data markup that helps search engines understand the content of articles. Benefits include: Rich snippets (date, author, publisher)…

Article schema is structured data markup that helps search engines understand the content of articles.

Benefits include:

  • Rich snippets (date, author, publisher)
  • Eligibility for Top Stories and carousels
  • Improved CTR and visibility

Common types: NewsArticle, BlogPosting, ScholarlyArticle. Implement using JSON-LD format for best compatibility with Google.

Breadcrumb Schema

Breadcrumb schema is a type of structured data (JSON-LD or Microdata) that helps search engines understand and display a site's breadcrumb navigation in the…

Breadcrumb schema is a type of structured data (JSON-LD or Microdata) that helps search engines understand and display a site's breadcrumb navigation in the SERPs.

By marking up breadcrumb paths with schema.org tags, you enable Google to show those breadcrumbs below your page title in search results, replacing long URLs.

This not only improves click-through rate but also provides better UX in search listings. Implementing breadcrumb schema is considered a best practice, especially for large or hierarchical websites.

It also enhances crawlability and site organization from an SEO perspective.

Crawl Budget

Crawl budget refers to the number of pages a search engine bot will crawl on your site within a given timeframe. It's influenced by two primary factors: crawl…

Crawl budget refers to the number of pages a search engine bot will crawl on your site within a given timeframe.

It's influenced by two primary factors: crawl rate limit (how often bots crawl without overloading your server) and crawl demand (how frequently content is updated or searched).

If your site has thousands of pages, ensuring that the most important ones are crawled and indexed becomes essential. You can optimize crawl budget by fixing broken links, removing duplicate content, enhancing internal linking, and updating your XML sitemap.

Sites with thin, irrelevant, or outdated content may waste crawl budget, leading to delays in indexing critical pages.

Crawler

A crawler (also known as a spider or bot ) is a script used by search engines to discover and index web content. It visits websites, follows links, reads…

A crawler (also known as a spider or bot) is a script used by search engines to discover and index web content.

It visits websites, follows links, reads content, and sends data back to the search engine’s servers. Google’s crawler is called Googlebot.

For SEO, it's vital to ensure that crawlers can access and understand your site’s content. This includes creating XML sitemaps, optimizing robots.txt files, and avoiding crawl errors.

Tools like Google Search Console and Screaming Frog help monitor crawl behavior and identify issues. If crawlers can’t reach your content, it won’t appear in search results.

Critical Rendering Path

The critical rendering path is the sequence of steps browsers take to convert HTML, CSS, and JavaScript into pixels on screen. Optimizing it means minimizing…

The critical rendering path is the sequence of steps browsers take to convert HTML, CSS, and JavaScript into pixels on screen.

Optimizing it means minimizing render-blocking resources, inlining critical CSS, and deferring non-essential scripts to improve perceived load speed.

Cumulative Layout Shift

Full guide

Cumulative Layout Shift (CLS) measures visual stability by tracking how much page elements move unexpectedly during load. A good CLS score is below 0.1. Common…

Read the full guide

Deindexed

A deindexed page is one that has been removed from a search engine’s index and no longer appears in search results. Deindexing can happen intentionally — using…

A deindexed page is one that has been removed from a search engine’s index and no longer appears in search results.

Deindexing can happen intentionally — using a noindex meta tag — or due to penalties, spammy content, malware, or violations of search engine guidelines.

Being deindexed is a serious issue for any website reliant on organic traffic. Recovery involves diagnosing the cause (manual penalty vs. technical issue), resolving it, and submitting the page for reindexing.

Tools like Search Console are essential for identifying deindexing events and requesting reconsideration.

Faceted Search Indexing

Faceted search indexing is the process of managing which filtered pages (from faceted navigation) are allowed to be indexed by search engines. Without controls,…

Faceted search indexing is the process of managing which filtered pages (from faceted navigation) are allowed to be indexed by search engines.

Without controls, faceted navigation can create millions of URL combinations, many with little to no SEO value — leading to crawl waste and index bloat.

SEO strategies for faceted indexing include:

  • Using canonical tags on default filter pages
  • Blocking excessive combinations with robots.txt or noindex
  • Managing crawl parameters in Google Search Console

Well-implemented faceted indexing supports both UX and crawl efficiency on large websites.

Google Index

The Google Index is the database of all web pages that Google has crawled and deemed valuable enough to serve in search results. Only indexed pages can rank.…

The Google Index is the database of all web pages that Google has crawled and deemed valuable enough to serve in search results.

Only indexed pages can rank. Pages may be excluded due to:

  • noindex tags
  • Crawl blocks
  • Low quality
  • Duplicates
  • Canonicalization conflicts

Use Google Search Console’s Coverage Report or URL Inspection Tool to check indexing status and submit new pages.

Proper indexing depends on crawlability, content quality, structured data, and sitemap inclusion.

Google Mobile-First Indexing

Mobile-first indexing means that Google predominantly uses the mobile version of a site for crawling and ranking. Since most users search on mobile, Google…

Mobile-first indexing means that Google predominantly uses the mobile version of a site for crawling and ranking.

Since most users search on mobile, Google prioritizes mobile UX and content parity. If your desktop and mobile versions differ significantly, the mobile version determines what gets indexed.

Key considerations:

  • Use responsive design
  • Ensure identical content and metadata across devices
  • Avoid blocked mobile resources

Tools like Search Console and PageSpeed Insights help monitor mobile performance and readiness.

Host Crawlability

Host crawlability refers to how well a website’s server responds to search engine bots attempting to access and crawl its pages. Poor server performance — like…

Host crawlability refers to how well a website’s server responds to search engine bots attempting to access and crawl its pages.

Poor server performance — like timeouts, 500 errors, or delayed responses — can prevent Googlebot from fully indexing a site.

Factors affecting crawlability include:

  • Server uptime
  • Bandwidth capacity
  • Page load speed
  • Proper handling of robots.txt

Tools like Google Search Console, server logs, and Screaming Frog help detect crawlability issues. Ensuring your host can support consistent crawling is foundational for sustained SEO visibility.

HTML Sitemap

An HTML sitemap is a human-readable page listing all (or most) of the pages on a website. Unlike XML sitemaps, which are meant for search engines, HTML sitemaps…

An HTML sitemap is a human-readable page listing all (or most) of the pages on a website.

Unlike XML sitemaps, which are meant for search engines, HTML sitemaps improve user navigation and help users find relevant content.

From an SEO perspective, they also assist crawlers in discovering deep or orphaned pages and contribute to internal link equity distribution.

Make sure your HTML sitemap is accessible from the footer and updates automatically with site changes.

Image Sitemap

An image sitemap is an XML sitemap that specifically lists image URLs on your website, helping search engines discover and index visual content more…

An image sitemap is an XML sitemap that specifically lists image URLs on your website, helping search engines discover and index visual content more efficiently.

This is especially useful for:

  • Ecommerce sites
  • Blogs with rich media
  • Portfolio or photography websites

Image sitemaps support attributes like caption, title, geo_location, and license.

Submit image sitemaps to Google via Search Console for better image SEO and visibility in Google Image search results.

Index Bloat

Index bloat occurs when a search engine indexes too many low-value, duplicate, or unnecessary pages from a site — diluting crawl efficiency and potentially…

Index bloat occurs when a search engine indexes too many low-value, duplicate, or unnecessary pages from a site — diluting crawl efficiency and potentially hurting SEO.

Symptoms include:

  • Indexed filter/sort URLs
  • Duplicate paginated content
  • Thin tag pages or archives
  • Parameter-based URLs

Solutions include:

  • Using noindex on low-value pages
  • Improving canonicalization
  • Cleaning up site architecture
  • Monitoring with Search Console and crawl tools

Managing index bloat ensures that only the most valuable content is prioritized for crawling and ranking.

Index Coverage Report

The Index Coverage Report in Google Search Console shows which pages of your website are indexed, excluded, or experiencing issues. It breaks down: Valid pages…

The Index Coverage Report in Google Search Console shows which pages of your website are indexed, excluded, or experiencing issues.

It breaks down:

  • Valid pages (indexed)
  • Errors (e.g., server or redirect issues)
  • Excluded pages (e.g., noindex, canonicalized, crawled but not indexed)

This report is critical for diagnosing crawl and indexing issues, understanding how much of your content is discoverable, and identifying areas to optimize.

Review it regularly to ensure your most important pages are being indexed and served properly in SERPs.

Indexed Content Ratio

The indexed content ratio is the proportion of a site’s total pages that are actually indexed by search engines. A low ratio (many pages not indexed) can…

The indexed content ratio is the proportion of a site’s total pages that are actually indexed by search engines.

A low ratio (many pages not indexed) can indicate crawl issues, thin content, or misconfigured robots.txt and meta tags.

To calculate:

Indexed pages / Total pages × 100

A healthy ratio suggests that your site's crawl budget is being used efficiently and that most content is considered valuable by search engines.

Regularly reviewing this metric helps prioritize content improvements and technical SEO fixes.

Indexed Pages

Indexed pages are the individual URLs from your website that have been crawled and stored in a search engine’s database. Only indexed pages are eligible to…

Indexed pages are the individual URLs from your website that have been crawled and stored in a search engine’s database.

Only indexed pages are eligible to appear in search results. The number of indexed pages should roughly match the number of valuable, crawlable pages on your site.

Too many indexed pages can indicate index bloat, while too few may suggest crawl or noindex issues.

Use Google Search Console’s Coverage Report to monitor indexing status and resolve issues like exclusions or errors.

Indexing

Indexing is the process by which search engines store and organize content found during crawling so that it can appear in search results. After Googlebot crawls…

Indexing is the process by which search engines store and organize content found during crawling so that it can appear in search results.

After Googlebot crawls a page, it analyzes the content, metadata, and links before deciding whether and how to include it in the index.

Only indexed pages can rank. If a page is not indexed, it’s invisible in search — even if it exists online.

You can check a page’s index status using the URL Inspection Tool in Google Search Console. Reasons for non-indexing include noindex tags, canonical conflicts, duplicate content, or crawl issues.

IndexNow

IndexNow is a protocol developed by Microsoft and adopted by search engines like Bing and Yandex (not Google — yet) that allows websites to notify search…

IndexNow is a protocol developed by Microsoft and adopted by search engines like Bing and Yandex (not Google — yet) that allows websites to notify search engines directly when content is created, updated, or deleted.

This speeds up indexing and ensures fresher results in search engines.

Sites using IndexNow can:

  • Instantly alert search engines to changes
  • Improve crawl efficiency
  • Reduce reliance on traditional bots

It’s implemented by generating an API key and pinging the IndexNow endpoint with URLs. CMS platforms like WordPress have plugins that automate this process.

Interaction to Next Paint

Full guide

Interaction to Next Paint (INP) is a Core Web Vitals metric that measures how quickly a page responds to user interactions like clicks and taps. A good INP is…

Read the full guide

Inverted Index

An inverted index is a data structure used by search engines to map keywords to the pages that contain them — rather than mapping pages to keywords. When a user…

An inverted index is a data structure used by search engines to map keywords to the pages that contain them — rather than mapping pages to keywords.

When a user types a search query, Google uses its inverted index to instantly find relevant documents that contain those terms.

Understanding this helps SEOs see why proper keyword usage and semantic structure matter — it improves how search engines retrieve and rank content.

While technical, it underpins the speed and precision of modern search algorithms.

JavaScript Crawl Budget

JavaScript crawl budget refers to the resources Google allocates to crawling and rendering JavaScript content on a site. JavaScript content is rendered later…

JavaScript crawl budget refers to the resources Google allocates to crawling and rendering JavaScript content on a site.

JavaScript content is rendered later than HTML, consuming more resources. If bots face inefficient scripts or long load times, they may not render or index content properly.

Minimizing render-blocking scripts, lazy-loading non-critical content, and prioritizing visible content are essential strategies.

Large, JS-heavy sites should optimize structure and rendering to avoid wasted crawl budget.

JavaScript Pagination Crawlability

Search engines often struggle to follow pagination implemented entirely through JavaScript if there are no crawlable links. To ensure crawlability: Use URLs…

Search engines often struggle to follow pagination implemented entirely through JavaScript if there are no crawlable links.

To ensure crawlability:

  • Use URLs that update with each page (e.g., ?page=2)
  • Include canonical tags to avoid duplication
  • Create HTML crawl paths or sitemaps
  • Avoid loading content only after user interaction (like scrolling)

Using dynamic rendering or pre-rendering is recommended for large, paginated JS-heavy sites.

Job Posting Indexation

Job posting indexation ensures that your job listings are discoverable by search engines and appear in Google for Jobs and related SERPs. To improve indexation:…

Job posting indexation ensures that your job listings are discoverable by search engines and appear in Google for Jobs and related SERPs.

To improve indexation:

  • Use job posting schema markup
  • Submit an XML sitemap specific to job URLs
  • Ensure job pages load fast and are mobile-friendly
  • Remove expired jobs promptly

Google requires job schema to follow its Job Posting content policies — non-compliance can lead to deindexing or reduced visibility.

Using Google Search Console’s Enhancements \> Job Posting section helps you monitor performance and fix issues.

Just-In-Time Indexing

Just-in-time indexing is an informal term describing the process where Google quickly indexes new or updated content in response to user demand or trending…

Just-in-time indexing is an informal term describing the process where Google quickly indexes new or updated content in response to user demand or trending queries.

It’s more likely to happen when:

  • The topic is trending
  • The domain has high authority
  • Content is submitted via Search Console

While not a guaranteed method, timely publication, structured data, and internal linking can speed up indexing and help content appear in news or top stories sections.

Largest Contentful Paint

Full guide

Largest Contentful Paint (LCP) measures how long it takes for the largest visible content element to load within the viewport. It is one of Google’s Core…

Read the full guide

Latent Semantic Indexing

LSI keywords are terms and phrases that are semantically related to a primary keyword. While Google has clarified it doesn't use LSI in the way SEOs often…

LSI keywords are terms and phrases that are semantically related to a primary keyword.

While Google has clarified it doesn't use LSI in the way SEOs often claim, the concept still reflects the importance of topic relevance and keyword diversity.

Using related terms helps search engines better understand content context and improves rankings for a broader range of queries.

Instead of focusing on LSI tools, use natural language, answer related questions, and cover a topic comprehensively to satisfy modern semantic search.

Meta Robots Tag

The meta robots tag instructs search engines on how to crawl and index a page. Common values include: index / noindex – whether to index the page follow /…

The meta robots tag instructs search engines on how to crawl and index a page. Common values include:

  • index / noindex – whether to index the page
  • follow / nofollow – whether to follow links on the page

Example:

<meta name="robots" content="noindex, nofollow">

These tags give granular control over how search engines interact with your content. Be cautious — incorrect usage (like noindex on important pages) can tank visibility.

Mobile-First Indexing

Full guide

Mobile-first indexing means that Google predominantly uses the mobile version of your site for crawling, indexing, and ranking. If your desktop and mobile…

Read the full guide

News Sitemap

A news sitemap is a specialized XML sitemap that helps Google discover and index news articles quickly. Best practices: Only include articles published in the…

A news sitemap is a specialized XML sitemap that helps Google discover and index news articles quickly.

Best practices:

  • Only include articles published in the last 48 hours
  • Limit to 1,000 URLs per sitemap
  • Use fields like <publication>, <title>, <access>, and <keywords>

News sitemaps are critical for publishers seeking inclusion in Google News or Top Stories. They increase crawl frequency and visibility for time-sensitive content.

Submit them in Google Search Console and ensure your site follows Google News content policies.

Noindex Tag

The noindex tag is a directive that tells search engines not to index a specific page. This prevents it from appearing in search results. It's added to the…

The noindex tag is a directive that tells search engines not to index a specific page. This prevents it from appearing in search results.

It's added to the &lt;head&gt; section of a page:

&lt;meta name="robots" content="noindex"&gt;

Common use cases:

  • Thank-you pages
  • Admin panels
  • Duplicate or thin content
  • Internal search results

While helpful for managing indexation, improper use on important pages can result in major ranking losses. Always test and monitor pages marked as noindex.

Non-Indexable Content

Non-indexable content refers to web pages or elements that search engines can’t add to their index. Common reasons include: noindex tags Disallowed by…

Non-indexable content refers to web pages or elements that search engines can’t add to their index.

Common reasons include:

  • noindex tags
  • Disallowed by robots.txt
  • Blocked by meta tags or HTTP headers
  • JavaScript rendering issues

Even if a page is crawlable, it may not be indexable.

Use tools like Search Console’s URL Inspection Tool or Screaming Frog to diagnose non-indexable pages and determine if adjustments are needed.

Product Schema

Product Schema is a type of structured data that provides search engines with detailed information about a product. It includes: Name Description Price SKU…

Product Schema is a type of structured data that provides search engines with detailed information about a product.

It includes:

  • Name
  • Description
  • Price
  • SKU
  • Availability
  • Review ratings

When implemented correctly using JSON-LD or Microdata, it can trigger rich snippets in search results — such as star ratings, pricing, and stock status — improving click-through rates and visibility in eCommerce search.

Review Schema

Review schema is a type of structured data that tells search engines a page includes reviews or ratings. Example: { "@type": "Review", "reviewRating": {…

Review schema is a type of structured data that tells search engines a page includes reviews or ratings.

Example:

{ "@type": "Review", "reviewRating": { "ratingValue": "4.5", "bestRating": "5" }, "author": { "@type": "Person", "name": "Jane Doe" } }

Implemented properly, it can trigger star ratings and review snippets in search results, improving CTR and credibility.

Applies to products, services, businesses, and even blog content when appropriate.

Robots Meta Tag

The robots meta tag controls how search engines index and follow individual pages. Example: &lt;meta name="robots" content="noindex, nofollow"&gt; Options…

The robots meta tag controls how search engines index and follow individual pages.

Example:

&lt;meta name="robots" content="noindex, nofollow"&gt;

Options include:

  • index / noindex — control visibility in search
  • follow / nofollow — control link crawling
  • noarchive — block cached versions
  • nosnippet — block snippets

Used with precision, it helps manage indexation and privacy. Misuse can block important pages from search results.

Robots.txt

The robots.txt file tells search engine crawlers which parts of your site to crawl or avoid. Example: User-agent: * Disallow: /private/ It’s placed in the root…

The robots.txt file tells search engine crawlers which parts of your site to crawl or avoid.

Example:

User-agent: * Disallow: /private/

It’s placed in the root directory (e.g., example.com/robots.txt) and affects crawl behavior — not indexing.

Be cautious: Misconfigured robots.txt can unintentionally block important pages or allow spam to be crawled.

Schema Markup

Full guide

Schema markup is a type of structured data added to a webpage’s code that helps search engines better understand its content. It uses a vocabulary called…

Read the full guide

Search Engine Index

The search engine index is a massive database of all the web pages that a search engine has crawled and deemed worthy of showing in search results. If a page is…

The search engine index is a massive database of all the web pages that a search engine has crawled and deemed worthy of showing in search results.

If a page is not indexed, it won’t appear in Google, no matter how optimized it is.

Pages can be excluded due to:

  • noindex tags
  • Crawl errors
  • Thin or duplicate content

Use Google Search Console or the site: operator to check if a page is indexed.

Signed Exchanges

Signed Exchanges (SXG) are a web technology that allows a site to be prefetched and displayed from a third-party cache while preserving the original…

Signed Exchanges (SXG) are a web technology that allows a site to be prefetched and displayed from a third-party cache while preserving the original publisher’s URL.

They can dramatically improve perceived page speed and are used by Google for Accelerated Mobile Pages (AMP) and prefetching.

Sitemap

A sitemap is a file that lists all URLs on a website, guiding search engine crawlers on how to discover and index pages. Two main types: XML sitemap – for…

A sitemap is a file that lists all URLs on a website, guiding search engine crawlers on how to discover and index pages.

Two main types:

  • XML sitemap – for search engines
  • HTML sitemap – for users

Best practices:

  • Submit to Google Search Console
  • Update regularly
  • Include only canonical, indexable URLs

A well-maintained sitemap supports SEO by ensuring full visibility of your site's important content.

Speculation Rules

Speculation Rules are a modern browser API that lets sites declaratively prefetch or prerender likely next-page navigations. When used carefully, they can make…

Speculation Rules are a modern browser API that lets sites declaratively prefetch or prerender likely next-page navigations.

When used carefully, they can make site navigation feel instant and improve engagement metrics without wasting bandwidth.

Technical SEO

Technical SEO focuses on optimizing a website’s infrastructure so search engines can effectively crawl, index, and rank content. It includes: Site speed Mobile…

Technical SEO focuses on optimizing a website’s infrastructure so search engines can effectively crawl, index, and rank content.

It includes:

  • Site speed
  • Mobile responsiveness
  • XML sitemaps
  • Canonical tags
  • HTTPS
  • Robots.txt
  • Structured data
  • Crawl budget optimization

Technical SEO ensures that great content can actually be discovered and understood by search engines. Regular audits using tools like Screaming Frog, Sitebulb, or Google Search Console are essential for maintaining site health.

Time to First Byte

Time to First Byte (TTFB) measures the time between a browser request and the first byte of the response arriving. It reflects server and network performance.…

Time to First Byte (TTFB) measures the time between a browser request and the first byte of the response arriving.

It reflects server and network performance. High TTFB can be caused by slow hosting, unoptimized databases, or excessive redirects.

Unindexed Pages

Unindexed pages are URLs that Google has crawled (or attempted to crawl) but has not included in its search index. Reasons may include: noindex tags Low-quality…

Unindexed pages are URLs that Google has crawled (or attempted to crawl) but has not included in its search index.

Reasons may include:

  • noindex tags
  • Low-quality or duplicate content
  • Crawl budget limits
  • Technical errors

Regularly audit index coverage in Search Console to spot and resolve unindexed issues, especially for critical pages.

Video Schema Markup

Video schema is a form of structured data that helps search engines understand and display video content in rich results. Common properties: Name and…

Video schema is a form of structured data that helps search engines understand and display video content in rich results.

Common properties:

  • Name and description
  • Thumbnail URL
  • Upload date
  • Duration
  • Embed URL

Adding video schema can trigger video carousels, rich snippets, and better indexation in Google’s video vertical.

Visibility Index

A visibility index is a score that indicates how visible a domain is across a range of tracked keywords in search engine results. Tools like Sistrix , Semrush ,…

A visibility index is a score that indicates how visible a domain is across a range of tracked keywords in search engine results.

Tools like Sistrix, Semrush, and Searchmetrics calculate this based on:

  • Keyword rankings
  • Search volumes
  • CTR-weighted visibility

It helps SEOs track overall performance beyond just traffic or rankings, showing trendlines for growth or decline across entire domains.

Web Crawler

A web crawler (also known as a spider or bot) is an automated program used by search engines to scan and index web pages. Popular examples: Googlebot (Google)…

A web crawler (also known as a spider or bot) is an automated program used by search engines to scan and index web pages.

Popular examples:

  • Googlebot (Google)
  • Bingbot (Bing)

Crawlers follow links, read content, and store information in the search engine’s index. Proper robots.txt configurations and XML sitemaps guide crawlers and ensure that only relevant content is indexed.

Controlling crawler access helps optimize crawl budget and site performance.

Website Indexing

Website indexing is the process by which search engines store web pages in their database so they can appear in search results. Only indexed pages are eligible…

Website indexing is the process by which search engines store web pages in their database so they can appear in search results.

Only indexed pages are eligible to rank. Indexing depends on:

  • Crawlability
  • Meta robots directives (index/noindex)
  • Site structure and internal links
  • Content uniqueness and value

You can inspect and request indexing via Google Search Console using the URL Inspection Tool.

X-Robots-Tag

The X-Robots-Tag is an HTTP header that controls indexing and crawling behavior of non-HTML resources like PDFs or images. Example: X-Robots-Tag: noindex,…

The X-Robots-Tag is an HTTP header that controls indexing and crawling behavior of non-HTML resources like PDFs or images.

Example:

X-Robots-Tag: noindex, nofollow

It functions similarly to meta robots tags but offers greater flexibility. It’s useful for preventing indexing of downloadable files or specific file types.

XML Sitemap

An XML sitemap is a structured file (usually sitemap.xml) that lists a website’s most important URLs for search engine crawling. Benefits: Guides crawlers to…

An XML sitemap is a structured file (usually sitemap.xml) that lists a website’s most important URLs for search engine crawling.

Benefits:

  • Guides crawlers to discover pages
  • Helps index new or updated content faster
  • Supports large or complex websites

Include canonical URLs only, keep the file under 50,000 URLs per sitemap (or 50MB), and submit it via Google Search Console for optimal crawling.

XML Sitemap Index

An XML Sitemap Index is a master sitemap file that links to multiple individual sitemaps — especially useful for large sites with thousands of URLs. Example:…

An XML Sitemap Index is a master sitemap file that links to multiple individual sitemaps — especially useful for large sites with thousands of URLs.

Example:

<sitemapindex> <sitemap><loc>https://example.com/sitemap1.xml</loc></sitemap> <sitemap><loc>https://example.com/sitemap2.xml</loc></sitemap> </sitemapindex>

This helps search engines efficiently discover all sitemap files, improving crawl coverage on large-scale domains.

Z-Index

Z-index is a CSS property that controls the stacking order of elements on a webpage. SEO relevance: Helps avoid content being hidden behind overlays Reduces…

Z-index is a CSS property that controls the stacking order of elements on a webpage.

SEO relevance:

  • Helps avoid content being hidden behind overlays
  • Reduces layout instability (CLS)
  • Improves mobile usability

Poor z-index settings can block important content or buttons, negatively affecting UX and accessibility.

Browse other categories

Put the theory to work

White-label link building and digital PR, priced per placement — no retainers required.

White-label links & AI search for agencies

Get a tailored proposal in 1 business day — no contracts.