
≈ 8 min read
article · 15 November 2026 · tools
Rules and a map.
Why a website needs robots.txt and sitemap.xml, which rules go into them, what to block from robots and what never to block, and how to check both files in Yandex Webmaster and Google Search Console.
Every website has two service files that visitors never open but search robots read all the time. Robots.txt contains the crawling rules: which sections the robot does not need to visit. Sitemap.xml is the site map: a list of the pages you want to show to the search engine. Both sit in the root of the site, and you can check whether you have them in a minute: open your-site.com/robots.txt and your-site.com/sitemap.xml in a browser.

Two files search engines read first
Every website has two service files that visitors never open but search robots read all the time. robots.txt contains the crawling rules: which sections the robot does not need to visit. sitemap.xml is the site map: a list of the pages you want to show to the search engine. Both sit in the root of the site, and you can check whether you have them in a minute: open your-site.com/robots.txt and your-site.com/sitemap.xml in a browser.
A business owner does not need to write these files - usually a developer or the CMS creates them. But it helps to understand what is inside: one wrong line in robots.txt can remove an entire services section from crawling, and an outdated site map can keep feeding the robot deleted pages for years. Below is what these files can do, what they cannot do, how to write robots.txt without dangerous mistakes and how to check both files.

What robots.txt is and how it works
robots.txt is an ordinary text file made up of a few lines of rules. Each rule starts with the name of a directive. The Yandex help pages list the supported ones:
- User-agent - which robot the rules below apply to.
User-agent: *means "all",User-agent: Yandexmeans Yandex robots only. - Disallow - what not to crawl.
Disallow: /cart/blocks the basket and everything that starts with that address. - Allow - what may be crawled even if the parent section is blocked.
- Sitemap - where the site map is, for example
Sitemap: https://your-site.com/sitemap.xml. - Clean-param - a Yandex directive: it tells the robot which URL parameters do not change the page content, so the same page with different tags is not treated as several pages.
Yandex understands two special characters: the asterisk * - any sequence of characters, and $ - the end of the address. Remember that a rule matches the beginning of an address: Disallow: /about blocks not only /about but also /about-us/ and anything else that starts the same way.
The requirements for the file itself are simple: it is called robots.txt, sits in the root of the site, returns a 200 status code and weighs no more than 500 KB. Cyrillic characters are not allowed - a Cyrillic domain is written in Punycode. If the file does not meet the requirements, Yandex treats the site as fully open to crawling.
The Host and Crawl-delay directives are not on the list of directives Yandex supports. If the robot puts too much load on the server, the crawl rate is set in Yandex Webmaster: "Indexing" → "Crawl rate".
What robots.txt cannot do
The most common misconception is that robots.txt removes pages from search. It does not:
- A blocked page can stay in search results. Yandex states plainly that such pages may remain in search. Google adds that a page blocked from crawling sometimes appears in results without a description - if other pages link to it.
- To remove a page from search, you need noindex - a meta tag in the page code or an HTTP header. And such a page must not be blocked in robots.txt at the same time: the robot has to visit it to see the instruction.
- It is not protection. The file is open to everyone, and its rules are a recommendation: major search engines follow them, but not every robot has to. Documents, contracts and personal accounts are protected with a password on the server, not with a line in robots.txt.
What to block and what to leave open
robots.txt exists so that the robot does not waste time on pages of no use to search. Typically blocked are:
- the basket, checkout and personal account;
- on-site search results;
- service sections and technical CMS folders;
- addresses with sorting, filter and ad tag parameters - if they create duplicates. How to organise catalogue filters, we covered in our article on SEO for an online store.
What must not be blocked: service and product pages, the sections you are promoting, and the style, script and image files without which the page looks different. Google's help pages advise against blocking such resources if their absence makes the page harder for the robot to understand.
A typical accident looks like this: the site was developed on a test address with the rule Disallow: /, and when it moved to the live domain nobody removed it. The whole site is blocked, and search traffic gradually drains away. That is why robots.txt is the first thing to check after launch and during a website redesign.
What sitemap.xml is and what goes into it
The sitemap.xml site map is a list of the addresses you consider important. It helps the search engine find new and deeply buried pages faster. If hardly any other pages of the site link to a page, the robot may not notice it for a long time - a site map and well-planned internal linking solve this problem together.
What matters about the format:
- Format - XML (recommended) or plain TXT, with one address per line. Encoding - UTF-8.
- Limits - no more than 50,000 addresses and 50 MB uncompressed per file. If there are more pages, the map is split into several files and they are combined in a Sitemap index file.
- Tags. The required one is
loc, the page address. The modification datelastmodis useful if it is accurate. Google states that it ignorespriorityandchangefreq; Yandex takespriorityinto account when deciding the order of crawling. - Your own addresses only - links in the map must point to the same domain where the file is located.
The map should contain only canonical pages that return a 200 status code: no redirects, no pages blocked in robots.txt or with noindex, no duplicates with tags. And most importantly, sitemap.xml does not guarantee indexing. Both Yandex and Google call it a hint: the system decides for itself whether to show a page in search.
In most CMSs the site map is generated automatically. Check that it updates when pages are added and does not contain deleted addresses.
How to check both files
- Yandex Webmaster: "Tools" → "robots.txt analysis". It shows syntax errors, keeps a history of changes to the file, and the "Are URLs allowed?" field lets you check whether a specific address is open.
- Site map in Webmaster: you can check the file with the "Sitemap file analysis" tool and submit it under "Indexing" → "Sitemap files" or with a line in robots.txt.
- Google Search Console: the settings include a robots.txt report - it replaced the old testing tool. The site map is submitted in the "Sitemaps" report.
Check robots.txt and the site map after every release and every move to a new CMS or hosting. The other Webmaster sections worth looking at regularly are gathered in our article on Webmaster for business owners. And if robots.txt and the site map are part of a larger review, it is easier to order it as a whole within SEO promotion.
In short
- robots.txt tells the robot where not to go; sitemap.xml tells it which pages matter.
- robots.txt sits in the root, returns a 200 code, weighs up to 500 KB and is written in Latin characters.
- Blocking a page in robots.txt does not remove it from search - that takes noindex.
- Do not block promoted pages, styles or scripts; check the file after every release.
- sitemap.xml should contain only canonical pages with a 200 code, up to 50,000 addresses per file.
- Both files can be checked in Yandex Webmaster and Google Search Console.
Frequently asked questions
Does a small website need robots.txt?
Without the file, search engines assume they may crawl everything. For a brochure site this is not a disaster, but a short robots.txt with a link to the site map and blocked service sections is still useful.
How do I remove a page from search?
Add a noindex meta tag to it or return an HTTP header that forbids indexing, and leave it open in robots.txt so the robot can see the instruction. If the page is no longer needed, you can delete it.
Do I need to update sitemap.xml by hand?
Usually not: the CMS or a plugin rebuilds the map when new pages appear. What is worth checking by hand is that the map really does update and contains no deleted addresses or redirects.
Can I block the whole site while it is in development?
You can, with the rule Disallow: /, but it is more reliable to protect the test version with a password. And be sure to remove the block on launch day.
articles
More equally interesting articles.

breakdowns
Website improvements without a rebuild
We look at which parts of a website can be improved separately, how to find weak spots, in what order to make changes and when targeted work no longer helps and a new project is due.
read the article15 November 2026ux / design
Consistent social media feed style
How to build a consistent visual style for your social media feed: choose a palette and fonts, prepare templates and covers, and keep enough variety so followers do not get bored.
read the article14 November 2026marketing
Retargeting: bringing visitors back
Which departed visitors are worth following up with ads, how to set up retargeting in Yandex Direct and VK Ads, and how to tell whether it brings sales rather than just cheap leads.
read the article14 November 2026FAQ
FAQ.
Yes, we offer full-cycle social media promotion: strategy, content creation, targeted advertising and results analytics.
We give you access to trello/asana where you can see every stage of the work. We also hold regular calls and send progress reports.
Yes, we work with staged payments. You pay for each completed stage according to the approved plan.
Timing depends on the project's complexity. On average a website takes 2 to 6 weeks. We always give exact deadlines before the work starts.
We can work with your content or offer our copywriters to create unique texts and select images.
The price is made up of the scope of work, functionality complexity, design and additional services (content, crm integration, etc.). We provide a detailed estimate broken down by stages.
With an audit: we study the business, the audience and the goals and shape a clear strategy. Then we fix the terms in a contract, agree on the details and start. Results are shown in numbers, and we build the project's growth on them.
You can leave a request at any time - our operators will call you back within a day. The office works on weekdays from 10:00 to 19:00 moscow time.
Still have questions? We may already know the answer
we'll get back to you within one business day

we help businesses grow
Ready to grow?
Leave a request and our managers will get back to you within one business day



