Robots.txt and indexing: why an allowed page can still render incorrectly

The pricing-page case

Sitelemetry's /pricing page returned HTTP 200 and was eligible for indexing, but Google's live rendering test could not show the price catalog correctly. The visible page depended on the public /api/plans response. A broader Disallow: /api/ rule also covered that dependency, so allowing the page itself was not enough.

On September 12, 2026, a narrow Allow: /api/plans$ exception was added. A subsequent Google live test showed the price cards and reported the page as indexable. An indexing request was then submitted. These observations support “the rendering obstruction was fixed.” They do not support “the page is now indexed” or any traffic-growth claim. No private account endpoint needed to be opened for this fix.

Ask three different questions

Can it be crawled? Robots rules tell supporting crawlers which paths they may fetch. Can its content be rendered? Scripts, styles and public data dependencies must also be available. Is it indexed? That is a separate search-engine decision, which a successful HTTP response cannot settle.

Google's robots.txt introduction explains that blocking crawling is not a dependable way to remove a URL from results. If an index exclusion is required, use an appropriate noindex directive and let the crawler fetch it; a robots block can prevent the directive from being seen. Read Google's noindex guidance before combining the two. Sensitive information must be protected by real access controls, independently of either mechanism.

Audit the page and its dependencies

1. Inspect the exact page URL, final redirect destination, response status and content type. Check both robots meta directives and relevant HTTP headers.
2. Read the robots file for the correct origin. Identify the applicable user-agent group and matching path rules instead of relying on the presence of an Allow: / line.
3. Compare delivered HTML with the meaningful content in a normal browser. List the script, style or public data requests that create missing sections.
4. Inspect the affected URL with Google's live test and review its rendered HTML, screenshot and reported resource failures where available.

A static check can reveal a suspicious rule, but it does not execute Google's renderer. Google's JavaScript SEO documentation describes crawling, rendering and indexing as separate stages. Use that model to decide what evidence is still missing.

Make the smallest justified change

First decide whether the blocked resource is truly public and required to understand an indexable page. In this case it was a public plan catalog, not customer data. The relevant rule fragment became:

User-agent: *
Disallow: /api/
Allow: /api/plans$

This is a focused example, not a replacement for an entire robots file. Google's matching uses the most specific applicable path; the $ marks an end match. The exception therefore should not be assumed to cover query variants or child paths. Check the exact requested URL against Google's robots specification.

Do not replace the broad block with permission for every API endpoint. Another option is to deliver essential public content in the initial HTML. Choose according to the page's dependencies and keep application authorization intact.

Retest what changed, not just the file

Verify that the updated robots file is publicly readable and contains the intended rules. Confirm that the necessary public resource still returns the expected data. Then repeat the live page test and inspect the actual content: are the price cards visible, not merely an empty container or loading message?

Record the before-and-after evidence and date. Check whether other resource failures remain. If the live test succeeds, an indexing request may be an appropriate next action, but keep monitoring the indexed result separately. A successful live test describes the current fetch and rendering conditions; it is not proof of completed indexing. Repeated requests do not create a guaranteed schedule. Report the corrected behavior precisely rather than turning an intermediate result into a ranking claim.

Choose the next SEO action from evidence

Apply this investigation first to pages whose missing content explains the product or supports a customer's decision. A broken pricing catalog deserves attention even before there is enough traffic to estimate its search impact. Afterward, review related templates for the same dependency pattern instead of changing unrelated rules speculatively.

Keep important public URLs discoverable through internal links and an accurate sitemap. Google's sitemap guidance makes clear that a sitemap helps discovery but does not guarantee crawling or indexing. Treat a sitemap, a correct response, a rendered page and an indexed URL as distinct pieces of evidence. Together they make a useful diagnostic trail; none promises impressions, clicks or purchases.