How search engines crawl, render and index a clinic website

Almost every conversation about clinic search performance starts in the wrong place. Someone asks why a page is not ranking for a treatment term when the more useful question is whether the page has completed the sequence that makes ranking possible at all. There are three separate stages between a page existing on a server and that page being eligible to appear on a results page, and each of them fails in its own way and for its own reasons.

Getting this order right is the single highest-value habit available to anyone who looks after a clinic website. It converts a vague problem, we are not visible, into a specific one that has a specific fix.

Stage zero: discovery

Before anything can be fetched, the URL has to be known. Search engines find URLs from links on pages they already crawl, from XML sitemaps, and from redirects that point at them. A page with no internal link pointing to it and no sitemap entry is not secret, it is simply undiscovered, and this is far more common on clinic sites than people expect. New treatment pages are frequently added to a content management system and linked only from a dropdown menu that is generated in the browser after page load, which means no crawler ever sees the link in the HTML.

The test is mechanical. Fetch your own page with JavaScript disabled and read the anchor tags in the source. If the link to the new page is not there, the page is not discoverable by that route. Sitemaps help, but a sitemap is a hint and not an instruction. Google's documentation on building a sitemap is explicit that inclusion does not guarantee crawling or indexing.

Stage one: fetch

Once a URL is known, the crawler requests it. Two things decide whether that request succeeds: your robots.txt file, and your server. Robots.txt is checked first, and it governs fetching only. A URL disallowed in robots.txt will not be fetched, but it can still be indexed on the strength of external links pointing at it, which is why robots.txt is the wrong tool for keeping a page out of search. The correct tool is a noindex directive on a page the crawler is allowed to fetch, because a directive on a page nobody may fetch is a directive nobody reads.

The parsing rules are standardised in RFC 9309, which is worth reading once because it removes a large amount of folklore about wildcards, ordering and specificity. The practical clinic failure looks like this: a staging site was blocked with a sitewide Disallow: /, the site went live, and the file was copied across with the block intact. Months of work is invisible and every report shows the same flat line.

Server behaviour matters as much. Intermittent 5xx responses, aggressive bot filtering on a firewall, and rate limiting all reduce how much of a site gets fetched. Clinic sites sitting behind a security product configured to challenge anything that is not a mainstream browser will frequently block legitimate crawlers by accident.

Stage two: parse and render

The fetched HTML is parsed. Links are extracted and added to the crawl queue, and content is extracted for indexing. If the page depends on JavaScript to build its own content, that content is not present at this point. It goes into a render queue, is executed later by a headless browser, and the results are then indexed. Google describes this two-pass behaviour in its JavaScript SEO documentation.

Two things follow from that. First, the delay between parse and render is not published and is not predictable, so anything critical that only exists after render is on an unbounded timer. Second, other crawlers are not obliged to render at all, and many do not. A clinic that renders its treatment descriptions client-side may be perfectly visible in Google and completely invisible to every other system that reads the web, including the crawlers that feed answer engines.

Stage three: indexing

Indexing is a decision, not an automatic consequence of a successful fetch. The system evaluates whether the page is worth storing, whether it duplicates something already stored, and which version of a set of near-identical URLs should be treated as canonical. A page can be crawled repeatedly and never indexed, and Search Console will say so in plain language in the Pages report: crawled, currently not indexed.

That status is usually a quality or duplication judgement rather than a technical fault. On clinic sites the most common cause is a family of treatment pages that differ only in the treatment name, with the same three paragraphs of introduction, the same call to action, and the same stock description of the consultation process. Google's guidance on creating helpful content is the right lens: if a page exists because a keyword exists rather than because a question exists, it will struggle.

Canonicalisation, the stage nobody watches

Where several URLs serve substantially the same content, one is selected as canonical and the rest are folded into it. You can express a preference with a rel="canonical" link, but it is a signal and not a command, and it competes with internal links, sitemap entries, redirects and duplicate content itself. Clinic sites generate duplicate URLs constantly: filtered listings, tracking parameters on paid campaigns, trailing slash variants, and both http and https or both www and non-www still resolving.

The cheapest fix in this whole article is to pick one hostname and one URL shape, redirect everything else to it with a permanent redirect, and make every internal link point at the final destination rather than at a redirect. Redirect chains do not usually stop indexing, but they waste crawl capacity and they are a reliable source of confusion when you are trying to work out what is actually indexed.

Only then, ranking

Ranking is downstream of all of the above and it is the part with the least published detail. What can be said with confidence is what the systems must do: match a query to documents, assess how well each document answers it, and assemble a results page from several surfaces at once. Anyone offering you a numeric weight for a specific ranking factor is describing a survey of opinion, not a measurement of the system, and the difference matters.

Working the sequence in practice

When a page underperforms, run the stages in order and stop at the first failure.

1. Is there an internal link to the URL in the server-rendered HTML, and is the URL in the sitemap?
2. Does robots.txt allow it, and does the server return 200 to a plain request?
3. Is the substantive content present with JavaScript disabled?
4. Does Search Console URL Inspection report it as indexed, and if not, what reason is given?
5. Is a different URL being treated as canonical instead of this one?
6. Does the page receive impressions for the queries you expected, or for none at all?

Impressions without clicks is a title and snippet problem. Clicks without enquiries is a page problem. No impressions at all is an indexing or relevance problem, and those are the ones this sequence is designed to catch before anyone spends a month rewriting copy that was never read.

How often any of this repeats

Recrawl frequency is not a setting and is not purchasable. It is inferred by the crawler from how often a URL changes and how important the site appears to be. Clinic sites are typically crawled slowly because they change slowly, which means a fix applied today may not be reflected for some time. Use URL Inspection to request indexing for genuinely important changes, treat it as a request rather than a lever, and resist the temptation to interpret a week of flat data as a failed fix.