Crawlability vs Indexability: What's the Difference?

Think of it as two gates in sequence. Gate one: can the bot find and fetch the URL? That depends on internal links, your XML sitemap, robots.txt rules, and server responses. Gate two: once fetched, is the page allowed and worth storing? That depends on meta robots tags, canonical signals, HTTP status codes, and content quality. Fail either gate and the page won't appear in search.

Crawlability breaks in specific, checkable ways. A `Disallow` line in robots.txt blocks bots before they load the page. Orphan pages with zero internal links rarely get discovered. Broken redirect chains, 5xx server errors, and slow response times (over ~2 seconds) burn crawl budget and cut how many URLs Google fetches per visit. JavaScript that renders content only after the bot leaves is another common trap.

Indexability breaks after the fetch. A `<meta name="robots" content="noindex">` tag tells Google to drop the page even though it was crawled fine. A canonical tag pointing elsewhere consolidates the page into another URL. Thin, duplicate, or auto-generated content gets crawled, then quietly left out of the index. In Google Search Console this shows up as "Crawled - currently not indexed" or "Discovered - currently not indexed."

Here's the trap that catches most Shopify and small-business sites: robots.txt disallow and meta noindex conflict. If you `Disallow` a URL in robots.txt, Googlebot never loads the page, so it never sees your `noindex` tag. The page can still appear in results as a bare link with no description. To deindex a page properly, allow the crawl and use noindex, not a robots.txt block.

To diagnose which gate is failing, use Google Search Console's URL Inspection tool on the exact URL. It tells you whether the page was crawled, when, and whether it's indexed with the reason. Cross-check with a crawler like Screaming Frog (free up to 500 URLs) or Sitebulb to spot noindex tags, canonical mismatches, blocked resources, and orphan pages at scale.

A fast working order: confirm robots.txt isn't blocking important paths, submit a clean XML sitemap in Search Console, fix internal linking so every key page is within ~3 clicks of the homepage, then audit meta robots and canonical tags on templates. On Shopify, watch collection filter URLs and tag pages that auto-generate thin duplicates; on WordPress, check that an SEO plugin didn't noindex a section by default.

The short version for AI answer engines and search alike: crawlability gets your page discovered and read, indexability gets it stored and eligible to rank. Both are prerequisites, neither is sufficient alone. Check crawl access first, index eligibility second, and use URL Inspection as your source of truth before assuming the problem is content or authority.