# How to Write a robots.txt File: Rules, Examples and Testing > How to write a robots.txt file for a business website: where it goes, rule syntax, safe examples, wildcards, sitemap lines and how to test and update it safely. URL: https://seoconsultantlondon.online/guides/how-to-write-a-robots-txt-file/ Updated: 2026-10-26 ## Quick answer A robots.txt file is a plain UTF-8 text file named robots.txt at the root of your host. Each group starts with a User-agent line followed by Allow or Disallow rules, and you can add Sitemap lines. Keep it short, block only low-value paths, test it in Search Console and never use it to hide private pages. ## Key takeaways - The file must be named robots.txt, sit at the root of the host it governs, and be plain UTF-8 text, with one file per protocol, host and port. - Each rule group starts with a User-agent line, followed by Allow or Disallow rules, and paths are case-sensitive and must start with a slash. - Wildcards and Sitemap lines are supported, and the wildcard user agent does not cover Google's AdsBot crawlers, which must be named explicitly. - Robots.txt manages crawling, not indexing, so it can't hide pages from search results and it isn't a security measure. - Test the file in a private window and in Search Console before and after every change, and keep a version history. ## What does a robots.txt file do, and what doesn't it do? Imagine a polite note taped to the door of your website. It says: "Crawlers, please don't go into the stockroom." Well-behaved visitors read it and comply. That's robots.txt in a sentence. It's a set of crawling instructions that reputable crawlers, including Googlebot, follow voluntarily. It's also one of the most misunderstood files on the internet. People expect it to hide pages, remove pages from Google, protect private content or boost rankings. It does none of those. What it does do is steer crawler traffic, which is useful when you have areas of the site that waste crawl effort or load your server for no benefit. > Why it matters: a good robots.txt is short and almost invisible. A bad one can block your whole site with a single line. ### What does Google say? Google's documentation, [Create and submit a robots.txt file](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt), last updated on 21 November 2025, sets out the rules. The file must be named robots.txt and placed at the root of the host it governs. A site can have only one. Subdomains and non-standard ports can have their own, and each file applies only to the protocol, host and port where it's posted. That last point catches people out. A robots.txt on the www host doesn't govern a blog on a separate subdomain, and a file served over HTTP doesn't automatically cover HTTPS. #### The purpose in plain terms The file tells crawlers where they may and may not go. It doesn't tell search engines what to index, and it doesn't tell them what to show. ### What is it not for? Keeping a page out of Google. The broader documentation on robots.txt is explicit that it isn't a mechanism for that: a blocked page can still appear in results, usually without a description, if other pages link to it. For removal, use noindex, password protection or deletion. The relationship between the two is explained in the guide to the [noindex tag](/guides/noindex-tag-when-to-use-it/). It's also not a security tool. Anyone can read your robots.txt, and bad bots ignore it. A line that says "Disallow: /private-offers/" advertises the folder's existence to anyone curious. If content must stay private, put it behind a login. #### The quick rule If you wouldn't mind the public reading the rule, robots.txt is fine for it. If you would, use proper access controls. ## Where does the file go and how must it be formatted? The technical basics are simple, and getting them wrong silently disables the file. A short checklist prevents most problems. Check these before you write a single rule. ### What are the file requirements? Name: exactly robots.txt, in lowercase. Location: the root of the host, so that https://www.example.co.uk/robots.txt returns it. Encoding: UTF-8 plain text, not a word-processor document. Count: one file per host. Access: it must return a 200 status and be publicly readable, which you can check in a private browsing window. Google's documentation warns specifically against using a word processor, since it can add characters that cause problems for crawlers. Use a plain text editor, or the editor built into your CMS or SEO plugin. | Requirement | Correct | Common mistake | |---|---|---| | File name | robots.txt | Robots.txt or robots.TXT | | Location | Root of the host | Inside /files/ or /seo/ | | Encoding | UTF-8 plain text | Word-processor formats | | Number per host | One | Several competing versions | | Scope | Per protocol, host and port | Assuming www covers subdomains | #### Check it publicly Open the URL in a private window. If you can read the rules there, so can crawlers. ### What if you use a CMS or plugin? Many platforms generate a virtual robots.txt automatically. WordPress, Shopify and similar systems serve one even though no physical file exists, and SEO plugins often provide an editor. Check what's being served before you create a physical file, because a physical file may override the virtual one, and two sources of truth cause confusion. If your platform lets you edit the file through a settings screen, use it, and keep a copy of the final text elsewhere. If it doesn't, ask your developer or hosting provider how to add or change it. #### Know where your file really comes from Plugin, platform or server file: find out which, and record it. The answer decides where you edit. ## How is a robots.txt file structured? The syntax is small enough to learn in ten minutes. A file is a series of groups, and each group is a user agent plus the rules for it. Here are the building blocks. ### What are user-agent groups? A group starts with a User-agent line naming the crawler the rules apply to, followed by one or more Disallow or Allow lines. The asterisk is a wildcard that matches all crawlers, with one notable exception: Google's documentation says it doesn't match the AdsBot crawlers, which must be named explicitly. You can have several groups. A crawler follows the most specific group that matches its name. If you don't need to treat any crawler differently, a single group for the wildcard user agent is plenty. #### A minimal example ``` User-agent: * Disallow: /search/ ``` This tells all crawlers not to crawl anything under /search/. ### How do Allow and Disallow work? Disallow tells crawlers not to fetch paths that match. Allow overrides a Disallow for a specific subfolder or page inside a blocked directory. Each rule sits on its own line. Paths must start with a slash, and directory paths should end with one. An empty Disallow means nothing is blocked. Google's page notes that rules are case-sensitive, so /Search/ and /search/ are different paths. It also confirms that a hash starts a comment, which is handy for documenting why each rule exists. #### Allow inside Disallow ``` User-agent: * Disallow: /cart/ Allow: /cart/help/ ``` Here the cart area is blocked, but the help page inside it can still be crawled. ### What about wildcards and sitemaps? All rules except the sitemap line support the asterisk, which matches any sequence of characters, for a prefix, suffix or whole string. A dollar sign can anchor a rule to the end of a URL, which is useful for file types. Sitemap lines take a fully qualified URL, and you can include several. Adding a Sitemap line is a low-risk, helpful habit, because crawlers that read robots.txt can discover your sitemap without a separate submission. The principles for the sitemap itself are in the guide to [XML sitemap best practices](/guides/xml-sitemap-best-practices/). #### Example with a sitemap ``` User-agent: * Disallow: /search/ Sitemap: https://www.example.co.uk/sitemap.xml ``` ## What should a small business robots.txt contain? Less than you think. A good file for a typical brochure or service site is a handful of lines. Every extra rule is a chance to block something by accident. Start with nothing and add only what you can justify. ### What are sensible rules to include? Block areas that offer no search value and generate endless URLs: internal search results, cart and checkout steps, account and login areas and admin sections. Allow everything else by default. Add the sitemap line. Add comments explaining each rule, so that future editors understand them. An example for a small shop might look like this: ``` # Keep crawlers out of low-value, endless or private-flow areas User-agent: * Disallow: /search/ Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Sitemap: https://www.example.co.uk/sitemap.xml ``` That's it. No clever patterns, no attempts to micro-manage crawling, no hiding. #### Don't copy someone else's file Another site's robots.txt reflects its own structure. Copying it blindly can block folders that matter on yours. ### What should you not block? CSS and JavaScript files that pages need to render, because blocking them can stop Google understanding pages properly. Your service, product and article pages. Your images if you want them found in image search. The sitemap file itself. And anything you want indexed, whatever you think of it. If you're tempted to block a section "to save crawl budget," pause. Most small sites don't have a crawl budget problem, and the explanation under [discovered, currently not indexed](/guides/discovered-currently-not-indexed/) shows that crawl issues usually have other causes. #### The default is open If you're unsure whether to block something, don't. You can always add a rule later. Removing the damage from a wrong rule takes longer. ## How do you test a robots.txt file? Writing the file is the easy part. Testing makes sure it does what you intended and nothing else. It takes five minutes, and it's the step most people skip. Google's documentation describes the approach. ### What should you check before you upload? That every rule is on its own line and starts with the right keyword. That paths start with a slash. That you haven't left a stray "Disallow: /" in place. That case matches your real URLs. That the file is saved as UTF-8 plain text. And that each rule has a reason you can explain. Read the file aloud, line by line, as a crawler would. "All crawlers: don't fetch anything under /search/." If any line surprises you, fix it before upload. #### Keep a copy of the old version Save the current file before replacing it. If something goes wrong, you can roll back in seconds. ### How do you test after you upload? Confirm the file is publicly accessible in a private browsing window. Then, as Google's page advises, check it with the robots.txt report in Search Console, which works for files already live on your site, or with Google's open-source robots.txt library, which you can run locally. In Search Console, open the robots.txt report to see which version Google fetched, when and whether there were parsing problems. Then test a sample of important URLs with the URL Inspection tool, confirming that "Crawl allowed?" says yes. The guide to [checking whether a page is indexed by Google](/guides/how-to-check-if-a-page-is-indexed-by-google/) shows how to read those fields. #### Test both allowed and blocked URLs Check five that should be crawlable and five that should be blocked. Both lists matter. ## How do you update robots.txt safely? Changing the file after launch carries more risk than writing it the first time, because there's a live site at stake. A short process keeps updates safe. Treat each change like a small release. ### What's a safe update process? Write down what you want to change and why. Edit a copy of the file. Test the new rules against sample URLs. Upload during a quiet period. Confirm the live file in a private window. Check the robots.txt report in Search Console. Then monitor the page indexing report and crawl stats over the following days. Google's documentation notes that to get its cached copy of the file refreshed sooner after an update, you can use the separate guide on submitting an updated file. Otherwise, Google refreshes it on its own schedule, so a change may take a little time to be picked up. #### Never edit directly on production without a backup A one-character mistake can block the whole site. A saved copy turns disaster into a thirty-second fix. ### Who should be allowed to change it? Name an owner, and limit editing rights. Many robots.txt accidents happen when a plugin, a developer or a hosting migration overwrites the file without anyone noticing. Add the file to your release checklist and set up a monitor that alerts you when its contents change. A small business doesn't need a complicated process. A named owner, a saved copy and a quick check after each release are enough to catch nearly every mistake. ## What mistakes break robots.txt files? A handful of errors cause most of the damage. Knowing them helps you spot a problem in a minute. Most are easy to avoid once you've seen them. ### Which syntax mistakes matter most? Leaving "Disallow: /" in place after launch. Forgetting the leading slash on a path. Mismatching case. Putting rules before the User-agent line. Using a word processor and picking up hidden characters. Writing a rule on the same line as another. Using a noindex line, which Google doesn't support in robots.txt. And serving the file from the wrong location or host. A tester or validator catches most of those. The ones that aren't syntax, such as blocking the wrong folder, need judgement and a sample test. #### The launch-day trap Staging sites often block everything. If the staging file is copied to production, Google is told to stay away from the whole site. Check the live file on launch day. ### Which strategic mistakes matter most? Blocking CSS and JavaScript that pages need. Using robots.txt to remove pages from the index. Blocking pages that you've also noindexed, so Google never sees the noindex. Blocking sections that have valuable links pointing to them. And relying on the file for privacy. The interplay between blocking and noindexing is explained in the guide to [pages blocked by robots.txt](/guides/blocked-by-robots-txt-search-console/), and it's worth reading before you decide how to remove a page from search. #### Ask what problem the rule solves If you can't state the problem in one sentence, the rule probably isn't needed. ## How do you write robots.txt for different site types? The principles are universal, but the sensible rules vary. A brief tour of common site types shows how to adapt. Use these as starting points, not templates to paste. ### What about ecommerce sites? Shops generate filter, sort and session URLs in huge numbers. Typical candidates for blocking are internal search results, cart, checkout and account paths and, with care, certain parameter patterns that create endless combinations. Be careful: blocking a filter pattern that has real search demand removes pages you may want to rank. Decide deliberately which combinations deserve to be crawled, as described in the guide to [faceted navigation and filter indexing](/guides/faceted-navigation-seo/), then block the rest. Test every pattern against real URLs. #### Keep product and category paths open Make sure no rule accidentally catches the folders that hold your products, categories or images. ### What about service and brochure sites? Usually the shortest files of all. Block internal search and any admin or utility areas, reference the sitemap and leave the rest alone. If your CMS creates thin archives, handle them with noindex or by turning them off, not with a long list of Disallow lines. If you have staging or development copies, protect them with a password. A robots.txt on staging is a courtesy, not protection, and it's the file most likely to be copied across to production by mistake. #### Multi-language and multi-location sites Keep the file simple and make sure every language folder and location page remains crawlable. Blocking one by accident is a classic cause of lost visibility abroad. ## How does robots.txt relate to sitemaps, noindex and canonicals? These tools work together, and they sometimes contradict each other. Understanding how they fit prevents the contradictions. A one-line summary of each helps. ### How do the tools divide the work? Robots.txt says where crawlers may go. Noindex says whether a page may appear. Canonical says which version is preferred. The sitemap says which URLs you'd like discovered. Redirects say where content has moved. Each answers a different question, and each works best when the others agree with it. A sitemap that lists a blocked URL, a noindexed page that's also blocked, or a canonical that points to a blocked URL are all contradictions. They create unpredictable results and confusing reports. The causes and fixes for sitemap conflicts are in the guide to [sitemap errors in Search Console](/guides/sitemap-errors-search-console/). #### The consistency check For any URL, ask: is it crawlable, indexable, canonical to itself and in the sitemap? If the answers don't match your intention, something is contradicting something else. ### What about crawl budget? For most small sites, robots.txt is not about crawl budget. Google's guidance is that crawl budget matters mainly for very large or fast-changing sites. For those, robots.txt can block unimportant URL patterns so crawling concentrates where it matters. For everyone else, the priority is accuracy, not optimisation. If you suspect a crawl problem, look at the data before you write rules. The Search Console crawl stats report shows what Googlebot is actually doing, and it's the right starting point for any decision. ## When should you get help with robots.txt? Most business sites can manage their own file. A few situations justify a second pair of eyes. Spotting them early saves a lot of grief. ### Which situations call for a specialist? A migration or redesign, where the file is easy to overwrite. A large or complex site with many sections and parameters. A platform that makes editing awkward. A sudden loss of indexed pages that might be rules-related. A history of conflicts between robots.txt, noindex and canonicals. Or any case where you can't explain why each rule exists. A [technical SEO consultant](/technical-seo-consultant/) can audit the file in the context of crawling, indexing and rendering, and a wider [SEO audit](/seo-audit/) shows how it interacts with the rest of the site. #### What a review should return A plain-English explanation of each rule, a list of risks, a simplified proposed file and a test plan. ### How do you keep it healthy? Add robots.txt to your launch checklist and your quarterly review. Keep the approved version in a shared document with comments. Set an alert for changes. And remind everyone who edits the site that a single line can make the whole thing invisible. A short, boring, well-commented file is a sign of a healthy site. If yours is long and clever, ask whether it needs to be. ## What does a worked example look like? An example brings the process together. This one is hypothetical, built from common patterns, with illustrative details. It follows a small business from muddle to a clean file. ### What was the situation? A small online shop had a robots.txt with forty lines, accumulated over several years by different developers and plugins. Some rules blocked folders that no longer existed. One blocked a category path that now held real products. Another blocked a script folder that product pages needed to render. Nobody knew why half of them were there. Search Console showed some product pages as blocked, and the owner suspected that was why new items took so long to appear. #### The discovery Reading the file line by line, with a spreadsheet of what each rule matched, showed that three rules were doing real damage and many were redundant. ### How was it fixed? The developer rewrote the file from scratch: block internal search, cart, checkout and account paths, point to the sitemap, and nothing else. They commented each line, tested ten product URLs and ten blocked URLs with the robots.txt report and URL Inspection, uploaded the file and confirmed it publicly. They saved the old version and the new one in the shop's documentation. Within a few weeks the blocked-by-robots count for products fell to zero and new items were discovered faster. The lesson is simple: a robots.txt file should be short enough to understand and tested enough to trust. Write it, check it, comment it and leave it alone unless there's a reason to change it. #### The takeaway Plain text, root of the host, a few well-chosen rules, a sitemap line and a test. That's all most websites need. ## How do you troubleshoot a robots.txt problem step by step? When something looks wrong, a calm sequence beats guesswork. Most robots.txt issues can be diagnosed in fifteen minutes with the tools you already have. Work from the outside in. ### What do you check first? Open the live file in a private window and confirm it loads, returns the content you expect and is the right version. Compare it with your saved copy. Check the robots.txt report in Search Console to see the version Google last fetched, when it was fetched and whether there were errors. If Google's copy differs from yours, the cached version may be out of date, so use the update route in Search Console. Then check whether the host is right. A file on the non-www host doesn't govern the www host. A file on HTTP doesn't automatically govern HTTPS. Many mysteries end here. #### Look for platform overrides If you edit a file and nothing changes, a plugin or platform may be serving a virtual file that overrides it. Find the real source before you edit again. ### How do you find which rule is blocking a URL? Take the affected URL and compare it with each rule, remembering that paths are case-sensitive and that wildcards match any sequence of characters. Use the URL Inspection tool to see whether crawling is allowed and, if you test locally with Google's open-source library, which rule matched. Once you know the rule, decide whether it should exist. If it blocks something valuable, narrow or remove it. If it does a useful job, add an Allow line for the exception. Retest, then record the change. | Symptom | Likely cause | First action | |---|---|---| | Whole site not crawled | A stray Disallow: / | Remove it, then request a recrawl of the file | | One section missing | A broad folder rule | Narrow the rule or add an Allow exception | | Pages render badly in the inspection tool | Blocked CSS or scripts | Allow the resource folders | | Edit has no effect | Virtual file or caching | Find the real source; refresh Google's copy | | Sitemap ignored | Wrong URL or blocked path | Fix the Sitemap line and check access | #### Don't panic-edit Make one change at a time, test it and write it down. Several simultaneous edits make it impossible to know what worked. ## How do you handle special crawlers and AI bots? Googlebot isn't the only visitor. Other search engines, SEO tools and AI crawlers also read robots.txt, and you may want different rules for some of them. A little care avoids unintended blocks. ### Can you set different rules for different bots? Yes. Each group starts with a User-agent line naming a crawler, so you can write one group for a specific bot and another for the wildcard. A crawler follows the most specific matching group. That means a rule for one named bot replaces, rather than adds to, the wildcard rules for that bot, which surprises people. Google's documentation notes that the wildcard doesn't match the AdsBot crawlers, which have to be named explicitly. If you run paid campaigns and block broad paths, check how those crawlers are affected, because landing pages need to be accessible to them. #### Test named groups separately If you add a named group, retest the URLs that matter against it. Specific groups replace general ones, and gaps can appear. ### What about blocking AI crawlers? Some site owners want to stop certain AI crawlers from collecting content. Many publish documented user-agent names and say they honour robots.txt, so you can add a group naming them with a blanket disallow. Check each provider's current documentation for the exact name and behaviour, because they change, and remember that compliance is voluntary. Think about the trade-offs before you block. Visibility in AI-driven answers may matter to your business, and blocking a crawler can reduce it. It's a business decision, not a technical default, and it's worth revisiting as the landscape changes. #### Keep a record of the decision Note which bots you block, why and when. Future you, or the next consultant, will want to know. ### How should you document the file for the next person? Add comments above each group explaining what it does and why it exists, with the date and the person who requested it. Keep a short change log alongside the file, listing every edit, the reason and the result of the test. Store both in the shared documentation where developers and marketers can find them. Good documentation turns a mysterious file into an understandable one. When a new consultant or developer joins, they can read the comments, check the log and see in minutes what the file is for, instead of guessing and risking a harmful edit. It also makes audits quicker, because every rule already carries its justification, and it makes the question "can we delete this line?" answerable with evidence instead of nervousness. #### Review the comments once a year Rules outlive the problems that created them. A yearly read-through often finds lines that can safely go, which keeps the file short and the risk low. ## Frequently asked questions ### Where do I put the robots.txt file? At the root of the host it governs, for example https://www.example.co.uk/robots.txt. A site can have only one robots.txt file per host, and each file applies only to the protocol, host and port where it's posted. Subdomains need their own file. It won't work inside a subfolder. ### What format must robots.txt use? Plain text encoded in UTF-8, named robots.txt. Avoid word processors, which can add characters that crawlers misread. Each group begins with a User-agent line, followed by Allow or Disallow rules, one per line. Paths must start with a slash, rules are case-sensitive, and a hash starts a comment. ### Do I need to submit robots.txt to Google? No. Google's crawlers find and use the file automatically once it's uploaded. If you update it and want Google's cached copy refreshed sooner, Search Console offers a way to request a recrawl of the file. Check the robots.txt report afterwards to see the version Google fetched. ### Can robots.txt hide a page from Google? No. Robots.txt controls crawling, not indexing, and a blocked URL can still appear in results if other pages link to it. To keep a page out of search, use noindex, password protection or removal. And don't use robots.txt for private content, because anyone can read the file. ### What happens if I don't have a robots.txt file? Crawlers assume they may crawl everything. That's fine for many small sites. A missing file returns a 404, which Google treats as no restrictions. You may still want a file to point to your sitemap and block low-value areas such as internal search results, but it isn't mandatory.