How to fix robots.txt and sitemap.xml before Google indexing
A valid XML file is not enough. Make the crawl rules, sitemap URLs, canonicals, index controls, and live responses agree before you ask a search engine to discover the site.
Serve a UTF-8 /robots.txt at the exact site root, remove accidental blocks, and point it to a valid sitemap. Put only absolute canonical URLs that return a useful page and are meant to be indexed in the sitemap. Then check each page for contradictory noindex, canonical, redirect, and status-code signals. Submit or inspect the sitemap in Search Console, but treat submission as a discovery hint rather than an indexing guarantee.
Work through the signals in order
Robots controls crawling. A sitemap helps discovery.
A robots.txt file tells compliant crawlers which paths they may request. It is mainly a crawl-traffic control file, not a reliable way to keep an ordinary web page out of search results. Google warns that a blocked URL can still be indexed from links without its page content being fetched.
A sitemap tells search engines which canonical URLs you consider important and when they changed. It can improve discovery, especially for a new site with few links, but it does not guarantee that a URL will be crawled, indexed, or ranked.
Indexing is decided from a wider set of signals: the live response, page content, canonical declaration, robots meta or header rules, links, duplicates, and search-engine judgment. That is why the repair order matters.
The crawlability preflight
| Signal | What to verify | Failure to avoid |
|---|---|---|
| robots.txt location | The file is available at the top level of the exact protocol and host being crawled, such as https://example.com/robots.txt. | A correct file on www does not govern the non-www host. |
| robots.txt format | Plain UTF-8 text, valid field/value lines, and comfortably below Google's 500 KiB processing limit. | An HTML error page, non-UTF-8 rules, or important directives after the size limit. |
| crawl rules | The user-agent group that applies to the crawler permits every page and resource it must fetch. | Disallow: /, overly broad wildcard paths, or blocked CSS/JS needed to understand the page. |
| sitemap format | Valid UTF-8 XML with entity-escaped values, at most 50,000 URLs and 50 MB uncompressed per file. | Malformed XML, relative URLs, mixed hostnames, or an oversized file. |
| sitemap membership | Each entry is the absolute canonical URL of a useful page you want indexed. | Redirects, 404s, duplicates, parameter variants, blocked pages, or noindex pages. |
| page response | The preferred URL returns a stable success response and useful content without a login wall. | Soft 404s, redirect chains, intermittent server errors, or empty client-only shells. |
| index controls | The page and HTTP headers do not carry an unintended noindex. | Blocking the page in robots.txt prevents the crawler from seeing a page-level noindex. |
| canonical | The declared canonical, internal links, sitemap URL, and redirects point to the same preferred URL. | A sitemap says one URL while rel="canonical" points elsewhere. |
| last modified | <lastmod> changes only after a significant page update and remains consistently accurate. | Stamping every URL with today's date on every build. |
A six-step repair workflow
Decide what should be indexed
Start with a list of real, canonical pages. Exclude private screens, internal search results, duplicate filters, broken pages, and thin placeholders. A sitemap should express an intentional indexable set, not dump every route the framework can generate.
Check the live root files
Request the production versions of /robots.txt and the sitemap URL. Verify the final URL, status, content type, body, and encoding. A local file can be perfect while the deployed route returns a redirect or branded 404.
curl -I https://example.com/robots.txt curl -I https://example.com/sitemap.xml curl -s https://example.com/robots.txt curl -s https://example.com/sitemap.xml
Read the applicable robots group
For each crawler you care about, find the most specific matching user-agent group and test the actual paths. Confirm that a broad rule is not swallowing a later exception. Keep the file small and explicit, and use a complete sitemap URL.
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
This is an example, not a universal prescription. A site may legitimately block low-value crawl paths while allowing its canonical pages.
Validate and cross-check the sitemap
Parse the XML, count entries, and inspect a sample plus every high-value URL. Follow redirects and compare each final page with its canonical and index controls. Split larger sitemaps or use a sitemap index before crossing the 50,000-URL or 50 MB uncompressed limit.
Use accurate <lastmod> values. Google says it may use them when they are consistently verifiable, while it ignores sitemap <priority> and <changefreq>.
Remove contradictory page signals
For every page that matters, align the live URL, HTTP status, canonical tag, robots meta tag, X-Robots-Tag, internal links, and sitemap entry. If a page must be excluded from results, let the crawler fetch its noindex rule; do not also hide that rule behind a robots block.
Submit, inspect, and wait for evidence
Expose the sitemap in robots.txt or submit it in Search Console. Inspect a representative high-value URL and the sitemap's settled status. Then monitor discovered, crawled, and indexed counts. A successful sitemap read confirms that Google could process the file; it does not promise that every listed page will enter the index.
Common contradictions that look like indexing problems
- Blocked and noindexed: the crawler cannot fetch the page, so it cannot read the
noindex. - Listed and redirected: the sitemap contains an old URL while the site consistently sends users elsewhere.
- Canonical mismatch: the sitemap lists one host or path but the page declares another canonical.
- Freshness theater: every
<lastmod>changes on every deploy even when the page content did not. - Discovery without value: the page is crawlable and technically indexable but duplicates another page or gives a searcher little reason to land there.
- Local-only validation: source files pass, but the CDN or application serves a different response in production.
Use the smallest check that answers the next question
Start by checking the local files without granting network access. Move to a live audit when the files are valid and you need to test real responses, rendering, performance, on-page signals, and competitor gaps.
robots.txt and sitemap files for malformed directives, broad blocks, missing sitemap references, bad URL schemes, mixed protocols, and file limits. Read-only and no network.
Check the local files →
SEO Auditor Pro
Move beyond the two discovery files with a live technical, on-page, Core Web Vitals, structured-data, and competitor-gap audit with prioritized fixes.
Audit the live site →
Primary sources
Start with the two files that decide discovery.
Run a local, read-only check before changing production. Then use a live audit for everything the files cannot prove.
Get the free validator on Agensi