Crawling and Indexing: From Basics to Understanding How Google Indexes Your Site

A ranking in search results comes down to two things at the bottom. Crawling, Google sends automated programs to your site and downloads the page content. Indexing, Google analyzes that content and stores it in its database, then picks the relevant pages to return when someone searches.

These two concepts are the most fundamental ideas in SEO. Before you dig into anything else, you have to understand them, because every optimization you run eventually works on one of these two stages. This article goes from basics to practice: first what each one is, then how Google actually runs them, and finally the common statuses and the troubleshooting order.

The facts here come mainly from Google’s official documentation. Before writing, I went back through the relevant pages in Search Central and Search Console Help, and I note the sources along the way. A few points come from the official IndexNow site, and I say so when we get there.

What crawling and indexing are

Google divides how Search works into three stages, crawling, indexing, and serving search results.

Crawling means Google’s automated programs, collectively known as Googlebot, download text, images, and videos from pages. Indexing means Google analyzes the page’s content and key tags and stores the information in the Google index, a large database. Serving search results means that when someone searches, Google picks relevant pages from the index and returns them.

Not every page makes it through all three stages. Before describing the stages, the official documentation adds a note: even if a page follows Google Search Essentials, Google doesn’t guarantee that it will crawl, index, or serve it.

Any team that treats indexing as a KPI should keep that sentence in mind. There is no quota and no promise for indexing; Google decides page by page based on its own assessment. The good news is that the mechanics are transparent. Following them significantly improves the odds of being indexed, and when you aren’t, you can see which step failed.

How Google discovers your pages

Google has to know a page exists before anything else. This step is called URL discovery, and there are only three entry points.

First, Google has visited the URL before. Redesigned pages and updated content fall into this bucket.

Second, links. When Google crawls a known page, it finds new URLs in the HTML links. A homepage linking to a new blog post counts, as do other websites linking to your pages.

Third, sitemaps. Google reads them regularly after you submit them in Search Console or declare their location in robots.txt.

Knowing a URL exists does not mean Google crawls it right away. Googlebot uses its own algorithms to decide which sites to crawl, how often, and how many pages to fetch at a time, and it throttles itself to avoid overwhelming sites. If a server keeps returning 500 errors, Google slows down.

Hard limits when Googlebot crawls

Googlebot is a generic name for two crawlers. Googlebot Smartphone simulates a user on a mobile device, and Googlebot Desktop simulates a desktop user. For most sites Google primarily indexes the mobile version of the content, so the majority of crawl requests come from the mobile crawler, with the desktop crawler in the minority.

A few hard parameters are easy to trip over. Googlebot only downloads the first 2MB of a supported file type, with one exception, PDFs, where it fetches the first 64MB. CSS and JavaScript referenced by the page are fetched separately, and each resource is subject to the same 2MB limit. Anything beyond the cutoff never reaches index evaluation. Put critical content very far down the page, or stuff the whole body into one giant script file, and Google may never see it.

On crawl frequency, the official line is that Googlebot shouldn’t access most sites more than once every few seconds on average. If your server can’t keep up, you can lower the crawl rate in Search Console. Google also throttles automatically when a server keeps returning 5xx or 429 responses.

If you want to confirm whether a visitor really is Googlebot, don’t trust the user-agent alone, it’s easily spoofed. The official recommendation is a reverse DNS lookup on the source IP, or matching the IP against Google’s published ranges.

robots.txt controls crawling, not indexing

The job of robots.txt is managing crawler traffic. The official documentation is blunt: it exists mainly to avoid overloading your site with requests. Keeping a page out of Google requires a noindex tag or password protection.

The effect also differs by file type. A blocked web page can still have its URL appear in search results. Blocked media files, images and videos, stay out of Google Images and Google Video results. So robots.txt is an effective way to keep images out of Google, but it’s unreliable for web pages. Another limit is syntax compatibility: robots.txt instructions depend on crawlers obeying them voluntarily, and different crawlers may interpret the syntax differently. It has no enforcement power, it’s a polite convention.

And then there’s the more counterintuitive part. If a page blocked by robots.txt has links pointing to it from elsewhere, Google may still index the URL. The address can appear in search results without a description, because Google never read the page content, but it knows the URL exists and it knows how others describe it. The official description of the “URL blocked by robots.txt” status in GSC says the same thing, this status does not guarantee the page won’t be indexed through some other means.

There’s an even sneakier trap. When a page is blocked by robots.txt, Google can’t read the noindex tag on it. Someone who thinks they’ve hidden a page thoroughly ends up with the opposite result: Google never sees the noindex and may index the URL anyway. Google’s Help documentation says explicitly that using a robots.txt rule to block a page is not recommended, because it actually prevents noindex from being seen by Google.

The conclusion is clear at this point. To control crawling, use robots.txt. To control indexing, use noindex or login protection. Don’t mix the two.

JavaScript pages take an extra rendering step

Many modern pages generate their content with JavaScript. When Google processes such pages, a rendering step sits between crawling and indexing, in the order crawling, rendering, indexing.

The renderer is a headless browser running an evergreen version of Chromium, a Chrome engine that stays updated to the latest version. Every page returning a 200 status enters the rendering queue, whether or not it contains JavaScript. The queue length isn’t fixed, a page may wait a few seconds, or longer. JavaScript-generated content naturally reaches the index half a step behind plain HTML.

Two practical takeaways. Google can discover links injected by JavaScript, as long as they’re standard a elements with href attributes. For single-page apps that switch content with hash routing, Google can’t reliably resolve those URLs, the History API works much better. Canonical declarations have their own nuance: Google recommends putting them in the original HTML, treating JavaScript as a fallback, and the value written by JavaScript must match the one in the HTML.

SSR and prerendering are still worth doing. Pages load faster for users, Googlebot processes them faster, and not all bots run JavaScript.

What size of site needs to care about crawl budget

Crawl budget is the number of URLs Google can and wants to crawl. The official definition has two parts, crawl capacity limit plus crawl demand. The capacity limit comes from what your server can take. Google gives every site a conservative starting value, raises it when the site responds steadily, and lowers it when 5xx or 429 errors appear. Demand comes from page popularity, update frequency, and content quality.

The concept only matters for sites of a certain scale. Google’s reference lines are these. More than one million unique pages with content changing about once a week. Ten thousand or more pages with content changing daily. Or a large share of URLs sitting in “Discovered - currently not indexed” in GSC. Google also notes these numbers are rough estimates. Small and medium sites can mostly ignore this; keeping the sitemap up to date and checking the Page Indexing report regularly is enough.

For large sites, the factor you can control most is your URL inventory. Duplicate pages, parameter-filtered pages, and infinite-scroll pages all consume crawl quota. Google recommends blocking such pages with robots.txt, and here the choice matters. Block crawling, don’t add noindex, because Google still crawls noindex pages, then drops them, wasting budget either way. For deleted pages, returning 404 or 410 is cleaner than blocking with robots.txt; Google gradually crawls them less, though it won’t forget the URL right away. One more line from the official docs. Google keeps recrawling known URLs that return 4xx for a while, and the only case where a URL won’t be crawled is when it returns a noindex directive.

Reading GSC statuses one by one

The Page Indexing report splits URLs into indexed and not indexed. Google reminds you not to expect every URL on your site to be indexed; the goal is to get the canonical version of every important page indexed. Duplicates and alternates not being indexed is usually a good thing, it means Google found the canonical and indexed it.

There’s one column that’s easy to overlook, Source. It marks whether an issue comes from the website or from Google. Issues marked Website are usually fixable on your side; issues marked Google usually have nothing to do with your site, leave them alone. Check this column before deciding whether a status needs action.

A few of the most common statuses.

“Crawled - currently not indexed” means the page was crawled but Google didn’t index it. It may or may not be indexed in the future, and Google says explicitly there’s no need to resubmit this URL. For these pages, look at the content itself: page quality, duplication, and the site’s overall assessment.

“Discovered - currently not indexed” means the page was found but not yet crawled. The typical reason in Google’s explanation: Google wanted to crawl the URL but the site was expected to be overloaded, so the crawl was rescheduled. On small and medium sites, this status most often means there are simply too many URLs and the page keeps waiting at the back of the crawl queue.

“Alternate page with proper canonical tag” and the two Duplicate statuses all describe normal operation. Duplicate pages, AMP versions, and mobile versions correctly point to the canonical page, and the canonical is indexed, nothing to do. A Duplicate status also tells you Google chose a canonical for you; if you disagree, check whether the canonical you declared makes sense.

“URL marked noindex” means the noindex tag is doing its job. If you don’t want the page indexed, fine. If you do, remove the tag.

“Page with redirect” is not an error by itself, the redirect target is what may get indexed.

“Soft 404” means the page returns 200 but reads as “not found”. Google treats it as a soft 404. Pages that are truly gone should return a real 404.

“URL blocked by robots.txt” we covered above, usually intentional, not an error. Confirm those pages really shouldn’t be read by Google; otherwise lift the block and switch to noindex.

Server-side problems show up as 5xx and assorted 4xx statuses. Those are real errors, fix the server, check the firewall and DNS.

Troubleshooting order and the actions you can take

When indexing problems hit, a good order is broad to narrow.

Look at the status itself first. Duplicate, Alternate, noindex, robots.txt blocks are often by design, don’t touch them.

Then look at the ratios. If the vast majority of Not indexed URLs are filter pages, parameter pages, and archive pages, the site is running normally. Then confirm just one thing: are your important pages indexed?

Then pick one specific URL and run it through the inspection tool. It tells you whether the crawl succeeded, whether rendering had problems, and which URL Google chose as canonical. After fixing something, you can request indexing there. Google says this can shorten the indexing lag for new content, but indexing is never instant, even when you submit a crawl request directly, and new content may take a few days to be processed.

The day-to-day actions are few. Keep the sitemap up to date, and add a lastmod tag for updated content. Return 404 or 410 for deleted pages. Consolidate duplicates. Keep the server stable and avoid 5xx. What remains after that is content quality and time.

For brand-new sites, add one more layer of expectation management. Google’s docs say it can take time, up to a few weeks, for Google to crawl some or all of a site once the URL is known. A new site with little indexing in its first week or two is usually normal; submit the sitemap and make the homepage and internal links solid first.

Validation timelines belong here too. When you click Validate fix in GSC, Google says validation typically takes about two weeks, sometimes much longer, during which Google recrawls the affected pages and notifies you by email. Indexing problems have no one-click fix.

About IndexNow and the Indexing API

IndexNow is a protocol that lets site owners actively notify search engines about content changes, a simple ping telling them a URL has been added, updated, or deleted. Its current supporters include Bing, Naver, Seznam, Yandex, and Yep. Google is not on the list.

Google’s official tool for fast indexing is the Indexing API, and its scope is narrow: it can only be used for two kinds of pages, job posting pages with JobPosting structured data, and livestream pages with BroadcastEvent embedded in a VideoObject. Regular content pages can’t use it.

For site owners targeting Google, counting on IndexNow or an API to speed up indexing points in the wrong direction. On Google’s side there are only two general entry points, sitemaps and requesting indexing.

Keep crawling and indexing separate, treat GSC statuses as clues rather than verdicts, and stretch the time horizon to weeks. With those in place, most indexing problems resolve on their own, and the small remainder is where the real fixes are.