Make the page easy to reach

The chain every page has to survive

Google describes Search as three stages. In crawling, automated programs download text, images and video from pages they have found. In indexing, Google analyzes the page and stores what it learned. In serving, it returns results that are relevant to a query.^[1] Each stage can stop a page, and each leaves different evidence. Rewriting a title will not help a page that answers with a server error, and a perfect sitemap will not help a page marked noindex.

Two details change how you debug. First, rendering sits between crawling and indexing. Googlebot queues pages that return a 200 status for rendering unless a robots rule tells it not to index them, and a headless Chromium then runs the JavaScript before Google indexes the rendered HTML.^[2] A page can be fetched without trouble and still be indexed without text that only JavaScript produces. The JavaScript chapter covers that case.

Second, nothing is guaranteed at any stage. Google says it does not guarantee that it will crawl, index or serve a page, even one that follows its guidelines.^[1] Your job is not to force inclusion. It is to remove every avoidable reason for exclusion and let the content earn the rest.

Robots.txt manages crawling, not indexing

A robots.txt file tells crawlers which URLs they may access. Google’s documentation says it exists mainly to avoid overloading your site and is not a mechanism for keeping a page out of Google. To keep a page out, block indexing with noindex or put the page behind a password.^[3] If a page is blocked in robots.txt but other pages link to it with descriptive text, Google can still index the bare URL without visiting it, and the result appears with no description.^[3]

That produces the most common contradiction in technical SEO. A team adds noindex to a page, then blocks the same page in robots.txt “to be safe.” The crawler can no longer fetch the page, so it never reads the noindex. Google states that robots meta rules can be read and followed only when crawlers are allowed to access the page that carries them.^[4]

Two more limits matter. Robots.txt is a convention honored by well-behaved crawlers and standardized as RFC 9309, which states that its rules are not a form of access authorization.^[5] Anything genuinely private needs authentication, and because robots.txt is a public file, a Disallow line also announces the path it protects. And changes are not instant: Google generally caches robots.txt for up to 24 hours and may keep it longer when it cannot refresh it.^[6]

# robots.txt for a small business site
User-agent: *
Disallow: /cart/
Disallow: /search
# Do not block /css/ or /js/: Google needs them to render pages.
Sitemap: https://example.com/sitemap.xml

<!-- on a page you want kept out of results (and still crawlable) -->
<meta name="robots" content="noindex">

Keep the file short. Block crawl-wasting areas such as carts and internal search, and leave alone any resource a page needs to render, since Google advises against blocking files whose absence makes a page harder to understand.^[3]

One page, one preferred address

The same content is often reachable at several URLs: with and without www, over http and https, with and without a trailing slash, and with tracking or sorting parameters appended. Search engines must decide which version to show, and signals such as links can be split across the copies. You can state your preference with three methods, listed here in the order Google says they influence the choice.^[7]

The methods stack, so using two or three raises the chance your preferred URL is chosen. None is required, and without a preference Google picks the version it judges best.^[7] It can also overrule your choice. The URL Inspection tool reports the Google-selected canonical for an indexed page, which is the quickest way to see whether your signals won.^[8]

https://example.com/services/sports-massage/            preferred
http://example.com/services/sports-massage/             301 to preferred
https://www.example.com/services/sports-massage/        301 to preferred
https://example.com/services/sports-massage/?utm=news   canonical to preferred

<link rel="canonical" href="https://example.com/services/sports-massage/">

Keep the signals consistent. Google’s guidance warns against naming one URL as canonical in the sitemap and another in the page, against using robots.txt for canonicalization and against using the URL removal tool for it, because that hides every version of the URL.^[7] When the report says Google chose a different canonical than you did, compare the two pages. Either they are near duplicates that should be merged, or your page does not yet differ enough to deserve its own URL.

Sitemaps are a hint, not a guarantee

A sitemap is a list of the URLs you want crawled. The rules are simple: a single file may hold up to 50,000 URLs or 50 MB uncompressed, URLs must be fully qualified and the file must be UTF-8.^[9] Google ignores the priority and changefreq values, and uses lastmod only when it has proven consistently accurate, so update it when the page meaningfully changes and not on every deploy.^[9] Submitting a sitemap does not guarantee anything; it is a hint.^[9]

The common mistake is a sitemap that contradicts the rest of the site. Search Console warns that error spikes can come from a submitted sitemap containing URLs blocked by robots.txt, marked noindex or behind a login.^[10] List only canonical, indexable pages that return 200. If your CMS generates the file, open it once and check that redirected, noindex and parameter URLs are absent.

<url>
  <loc>https://example.com/services/sports-massage/</loc>
  <lastmod>2026-09-14</lastmod>
</url>

For engines other than Google there is also IndexNow, a protocol that pings participating search engines when a URL is added, updated or deleted. Its project site lists Microsoft Bing, Naver, Seznam.cz, Yandex and Yep as supporters and says that without it, discovering a change can take days to weeks.^[11] Google is not among the supporters named there, so a sitemap and good internal links remain the baseline.

Read Search Console as a diagnostic, not a scoreboard

The Page indexing report starts from a realistic premise. It says most sites have at least a few unindexed pages, that all important pages should be indexed and that you should not expect every URL to be.^[10] Duplicates, blocked utility pages and removed pages belong in the not-indexed column. Triage by importance: start with the pages that carry enquiries, and ignore the rest until those are healthy.

Use the URL Inspection tool for single pages. It shows the indexed version of a URL, lets you test the live version, renders a screenshot of how Google’s inspection tool sees the page and lists the resources and JavaScript output it loaded.^[8] After a fix, request indexing and note the date. Google’s starter guide says changes can take from a few hours to several months to show, and suggests waiting a few weeks before judging the result.^[12]

Work a failure from symptom to fix

When a page is missing, walk the chain in order and stop at the first failure. Each check is cheap, and an early fix often explains everything after it.

Handle moves and removals on purpose

Most indexing damage on established sites comes from changing URLs without a plan. Decide what each old address should do, then make the server say so.

Keep redirect chains short. By default Google’s crawlers follow up to ten redirect hops, but every extra hop is another request that can fail, so point each old URL straight at its final destination.^[15] Before a launch, crawl the staging site once and compare its URL list with the old one. The gaps are your redirects.

Apply it to your website

• Each important URL ends in a 200 response after at most one redirect, and the final address matches your preferred version.
• robots.txt blocks only low-value areas such as carts and internal search, and never the CSS or JavaScript a page needs to render.
• Pages you want kept out of results carry noindex and stay crawlable, or sit behind authentication. None relies on robots.txt alone.
• Every indexable page names one canonical URL, and redirects, canonicals, internal links and the sitemap all agree.
• The sitemap lists only canonical, indexable pages that return 200, and lastmod changes only when the content does.
• Your pages that matter appear as Indexed in the Page indexing report, and you can explain each status in the not-indexed list.
• After every fix you note the date and the change, then judge the result after a few weeks rather than the next day.