Retrieval eligibility begins with access. Before any question about content quality is worth asking, it is worth knowing which systems are permitted to fetch your pages, which are blocked, and whether either state was chosen on purpose.
What robots.txt does
Robots.txt is a plain text file at the root of a host that tells compliant crawlers which paths they may fetch. Its parsing behaviour is standardised in RFC 9309. Three properties matter and are constantly confused.
- It controls fetching, not indexing. A disallowed URL can still be indexed from external links, without a description.
- It is voluntary. Compliant crawlers obey it. Nothing enforces it technically.
- It is public. Anyone can read it, including whoever you were hoping to hide a path from.
The agents that request clinic pages
Several categories of crawler will appear in your logs.
| Category | Purpose | Consequence of blocking |
|---|---|---|
| Search crawlers | Building the web search index | You leave search entirely |
| AI training crawlers | Collecting data for model training | No effect on search indexing |
| Grounding and retrieval fetchers | Fetching pages to answer a live question | You become uncitable in that product |
| Preview and link unfurl bots | Generating link previews | Your links look broken when shared |
| Commercial SEO crawlers | Third-party link and rank databases | You disappear from their data |
The distinction between the second and third rows is the one that matters most and the one most often missed. Blocking a training crawler is a decision about your content being used to train a model. Blocking a live retrieval fetcher is a decision to be absent from answers in that product. They are different decisions with different consequences and they are frequently made together by accident.
Named agents and where to find them
Google publishes a complete list of its crawlers and user agents, including the separate token used to control use of content for its generative products independently of search indexing. Other major providers publish their own agent names and their documented behaviour. Use the published lists rather than a blog post, because agent names change and the published list is the only version that is current by definition.
Many clinic sites sit behind a security or performance layer that challenges any request that does not look like a mainstream browser. That configuration blocks legitimate crawlers at the network level, before robots.txt is ever consulted, and it is invisible unless you look at logs or test with a custom user agent. Check this before concluding anything about your content.
AI answers and assistants
LOW CONTROL- Whether your content is crawlable by the relevant user agents
- How clearly a page states the fact you want quoted, and where
- Whether the same fact is consistent across your site, profile and listings
- Structured data that removes ambiguity about who and what you are
- Whether a model chooses to cite you
- Which passage it lifts, or how it paraphrases it
- Whether an answer appears at all for a given prompt
- The training data of any model already released
- Run the same prompt repeatedly and record what is cited, with dates
- Check server logs for AI user agents rather than guessing at access
- Verify the fact you want quoted appears verbatim on a crawlable page
Making the decision
For a typical clinic the reasoning is straightforward. Your published content is marketing and patient information. It exists to be found and quoted. Blocking retrieval fetchers removes you from an increasingly used discovery surface in exchange for nothing, because the content was public anyway.
The argument for blocking training crawlers is different and can be legitimate: original research, proprietary methodology or photography you license. Most clinics have none of that on the website, and where they do, it is usually better protected by not publishing it than by a robots directive nobody is obliged to obey.
A workable clinic robots.txt
User-agent: * Allow: / Disallow: /search Disallow: /*?s= Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # Named agents are listed explicitly so the decision is visible to # anyone reading the file, including whoever maintains it next. User-agent: Googlebot Allow: / User-agent: Google-Extended Allow: / User-agent: Bingbot Allow: / Sitemap: https://example.co.uk/sitemap.xml
Note what this file does not do. It does not attempt to hide anything sensitive, because robots.txt is public and voluntary. Anything genuinely confidential belongs behind authentication, not behind a disallow line.
Verifying the state you think you are in
- Fetch
/robots.txtand read it. Not the version in the plugin: the served file. - Request a page with a custom user agent and check the status code. If it is challenged, your security layer is the control, not robots.txt.
- Search your access logs for known crawler agents and confirm they receive 200 responses.
- Check for an
X-Robots-Tagheader, which can override page-level expectations invisibly. - Re-check after any change of host, CDN or security product, because these settings reset.
Rate limiting and load
Crawler volume can be significant on a small site. If load is a genuine problem, address it with caching and with the rate controls your CDN offers rather than with blanket blocks, because a block is permanent absence and a rate limit is a slower welcome.
Review it annually
Agent names change, new products appear, and infrastructure gets replaced. Put an annual review in the calendar: read the served robots.txt, check the published agent lists, confirm the access decision still matches what the clinic wants, and confirm the logs agree with the file. Ten minutes, once a year, prevents the situation where a site has been invisible to a whole class of reader since a hosting migration nobody documented.