≈ 8 min read

article · 15 November 2026 · tools

Rules and a map.

Why a website needs robots.txt and sitemap.xml, which rules go into them, what to block from robots and what never to block, and how to check both files in Yandex Webmaster and Google Search Console.

written by

Evgeny

Evgeny

developer · 14 years

topic

Tools

date

15 November 2026

Every website has two service files that visitors never open but search robots read all the time. Robots.txt contains the crawling rules: which sections the robot does not need to visit. Sitemap.xml is the site map: a list of the pages you want to show to the search engine. Both sit in the root of the site, and you can check whether you have them in a minute: open your-site.com/robots.txt and your-site.com/sitemap.xml in a browser.

Two files search engines read first

Every website has two service files that visitors never open but search robots read all the time. robots.txt contains the crawling rules: which sections the robot does not need to visit. sitemap.xml is the site map: a list of the pages you want to show to the search engine. Both sit in the root of the site, and you can check whether you have them in a minute: open your-site.com/robots.txt and your-site.com/sitemap.xml in a browser.

A business owner does not need to write these files - usually a developer or the CMS creates them. But it helps to understand what is inside: one wrong line in robots.txt can remove an entire services section from crawling, and an outdated site map can keep feeding the robot deleted pages for years. Below is what these files can do, what they cannot do, how to write robots.txt without dangerous mistakes and how to check both files.

Hands unfolding a paper floor plan of a building on a desk, with no lettering

What robots.txt is and how it works

robots.txt is an ordinary text file made up of a few lines of rules. Each rule starts with the name of a directive. The Yandex help pages list the supported ones:

  • User-agent - which robot the rules below apply to. User-agent: * means "all", User-agent: Yandex means Yandex robots only.
  • Disallow - what not to crawl. Disallow: /cart/ blocks the basket and everything that starts with that address.
  • Allow - what may be crawled even if the parent section is blocked.
  • Sitemap - where the site map is, for example Sitemap: https://your-site.com/sitemap.xml.
  • Clean-param - a Yandex directive: it tells the robot which URL parameters do not change the page content, so the same page with different tags is not treated as several pages.

Yandex understands two special characters: the asterisk * - any sequence of characters, and $ - the end of the address. Remember that a rule matches the beginning of an address: Disallow: /about blocks not only /about but also /about-us/ and anything else that starts the same way.

The requirements for the file itself are simple: it is called robots.txt, sits in the root of the site, returns a 200 status code and weighs no more than 500 KB. Cyrillic characters are not allowed - a Cyrillic domain is written in Punycode. If the file does not meet the requirements, Yandex treats the site as fully open to crawling.

The Host and Crawl-delay directives are not on the list of directives Yandex supports. If the robot puts too much load on the server, the crawl rate is set in Yandex Webmaster: "Indexing" → "Crawl rate".

What robots.txt cannot do

The most common misconception is that robots.txt removes pages from search. It does not:

  • A blocked page can stay in search results. Yandex states plainly that such pages may remain in search. Google adds that a page blocked from crawling sometimes appears in results without a description - if other pages link to it.
  • To remove a page from search, you need noindex - a meta tag in the page code or an HTTP header. And such a page must not be blocked in robots.txt at the same time: the robot has to visit it to see the instruction.
  • It is not protection. The file is open to everyone, and its rules are a recommendation: major search engines follow them, but not every robot has to. Documents, contracts and personal accounts are protected with a password on the server, not with a line in robots.txt.

What to block and what to leave open

robots.txt exists so that the robot does not waste time on pages of no use to search. Typically blocked are:

  • the basket, checkout and personal account;
  • on-site search results;
  • service sections and technical CMS folders;
  • addresses with sorting, filter and ad tag parameters - if they create duplicates. How to organise catalogue filters, we covered in our article on SEO for an online store.

What must not be blocked: service and product pages, the sections you are promoting, and the style, script and image files without which the page looks different. Google's help pages advise against blocking such resources if their absence makes the page harder for the robot to understand.

A typical accident looks like this: the site was developed on a test address with the rule Disallow: /, and when it moved to the live domain nobody removed it. The whole site is blocked, and search traffic gradually drains away. That is why robots.txt is the first thing to check after launch and during a website redesign.

What sitemap.xml is and what goes into it

The sitemap.xml site map is a list of the addresses you consider important. It helps the search engine find new and deeply buried pages faster. If hardly any other pages of the site link to a page, the robot may not notice it for a long time - a site map and well-planned internal linking solve this problem together.

What matters about the format:

  • Format - XML (recommended) or plain TXT, with one address per line. Encoding - UTF-8.
  • Limits - no more than 50,000 addresses and 50 MB uncompressed per file. If there are more pages, the map is split into several files and they are combined in a Sitemap index file.
  • Tags. The required one is loc, the page address. The modification date lastmod is useful if it is accurate. Google states that it ignores priority and changefreq; Yandex takes priority into account when deciding the order of crawling.
  • Your own addresses only - links in the map must point to the same domain where the file is located.

The map should contain only canonical pages that return a 200 status code: no redirects, no pages blocked in robots.txt or with noindex, no duplicates with tags. And most importantly, sitemap.xml does not guarantee indexing. Both Yandex and Google call it a hint: the system decides for itself whether to show a page in search.

In most CMSs the site map is generated automatically. Check that it updates when pages are added and does not contain deleted addresses.

How to check both files

  • Yandex Webmaster: "Tools" → "robots.txt analysis". It shows syntax errors, keeps a history of changes to the file, and the "Are URLs allowed?" field lets you check whether a specific address is open.
  • Site map in Webmaster: you can check the file with the "Sitemap file analysis" tool and submit it under "Indexing" → "Sitemap files" or with a line in robots.txt.
  • Google Search Console: the settings include a robots.txt report - it replaced the old testing tool. The site map is submitted in the "Sitemaps" report.

Check robots.txt and the site map after every release and every move to a new CMS or hosting. The other Webmaster sections worth looking at regularly are gathered in our article on Webmaster for business owners. And if robots.txt and the site map are part of a larger review, it is easier to order it as a whole within SEO promotion.

In short

  • robots.txt tells the robot where not to go; sitemap.xml tells it which pages matter.
  • robots.txt sits in the root, returns a 200 code, weighs up to 500 KB and is written in Latin characters.
  • Blocking a page in robots.txt does not remove it from search - that takes noindex.
  • Do not block promoted pages, styles or scripts; check the file after every release.
  • sitemap.xml should contain only canonical pages with a 200 code, up to 50,000 addresses per file.
  • Both files can be checked in Yandex Webmaster and Google Search Console.

Frequently asked questions

Does a small website need robots.txt?

Without the file, search engines assume they may crawl everything. For a brochure site this is not a disaster, but a short robots.txt with a link to the site map and blocked service sections is still useful.

How do I remove a page from search?

Add a noindex meta tag to it or return an HTTP header that forbids indexing, and leave it open in robots.txt so the robot can see the instruction. If the page is no longer needed, you can delete it.

Do I need to update sitemap.xml by hand?

Usually not: the CMS or a plugin rebuilds the map when new pages appear. What is worth checking by hand is that the map really does update and contains no deleted addresses or redirects.

Can I block the whole site while it is in development?

You can, with the rule Disallow: /, but it is more reliable to protect the test version with a password. And be sure to remove the block on launch day.

FAQ

FAQ.

Yes, we offer full-cycle social media promotion: strategy, content creation, targeted advertising and results analytics.

We give you access to trello/asana where you can see every stage of the work. We also hold regular calls and send progress reports.

Yes, we work with staged payments. You pay for each completed stage according to the approved plan.

Timing depends on the project's complexity. On average a website takes 2 to 6 weeks. We always give exact deadlines before the work starts.

We can work with your content or offer our copywriters to create unique texts and select images.

The price is made up of the scope of work, functionality complexity, design and additional services (content, crm integration, etc.). We provide a detailed estimate broken down by stages.

With an audit: we study the business, the audience and the goals and shape a clear strategy. Then we fix the terms in a contract, agree on the details and start. Results are shown in numbers, and we build the project's growth on them.

You can leave a request at any time - our operators will call you back within a day. The office works on weekdays from 10:00 to 19:00 moscow time.

Still have questions? We may already know the answer

we'll get back to you within one business day

we help businesses grow

Ready to grow?

Leave a request and our managers will get back to you within one business day

Discuss a project.

your contacts
what is the task?