Learn how to include your XML sitemap in the robots.txt file for better search engine crawling and indexing.
When it comes to optimizing your website for search engines, ensuring that search engine bots (robots) can efficiently crawl and index your pages is crucial. Two essential files play a significant role in this process: robots.txt and XML sitemaps. In this article, we’ll explore how to add your sitemap reference to the robots.txt file, enhancing your site’s discoverability and SEO performance.Sitemaps tell Google which pages on your website are the most important and to be indexed. While there are many ways to create a sitemap, adding it to robots.txt is one of the best ways to ensure that it is seen by Google.
Understanding Robots.txt
What Is Robots.txt?
The robots.txt file is a plain text file placed in your website’s root directory. Its purpose is to provide instructions to search engine robots regarding which pages they can or cannot crawl. Even if you want all robots to access every page on your site, having a robots.txt file is good practice.
The Role of XML Sitemaps
An XML sitemap is an XML file listing all the pages you want search engine bots to discover and access. It’s essential for ensuring comprehensive coverage of your site. By including your sitemap in the robots.txt file, you guide search engines to the right places.
How Are robots.txt & Sitemaps Related?
Back in 2006, Yahoo, Microsoft and Google united to support the standardized protocol of submitting a website’s pages via XML sitemaps. You were required to submit your XML sitemaps through Google Search Console, Bing webmaster tools and Yahoo, while some other search engines such as DuckDuckGoGo uses results from Bing/Yahoo.
After about six months, in April 2007, they joined in support of a system to check for XML sitemaps via robots.txt, known as Sitemaps Autodiscovery.
This meant that even if you did not submit the sitemap to individual search engines, it was OK. They would find the sitemap location from your site’s robots.txt file first.
(NOTE: Sitemap submission is still available through most search engines, but don’t forget, Google & Bing aren’t the only search engines!)
And hence, the robots.txt file became even more significant for webmasters because they can easily pave way for search engine robots to discover all the pages on their website.
How To Add Your XML Sitemap To Your Robots.txt File
Here are three simple steps to adding the location of your XML sitemap to your robots.txt file:
Step 1: Locate Your Sitemap URL
If your website has been developed by a third-party developer, you need to first check if they provided your site with an XML sitemap.
By default, the URL of your sitemap will be /sitemap.xml. For example, the XML sitemap for https://befound.pt is
So type this URL in your browser with your domain in place of ‘befound.pt’.
Some websites have more than one XML sitemap, which requires a sitemap for sitemaps (known as a sitemap index). For example, if you’re using the Yoast SEO plugin with WordPress, a sitemap index will automatically be added to /sitemap_index.xml.
You may also be able to locate your sitemap via Google search by using search operators as shown in the examples below :
site:befound.pt filetype:xml
OR
filetype:xml site:befound.pt inurl:sitemap
But this will only work if your site is already crawled and indexed by Google.
If you have access to your website’s File Manager, you can search for your xml sitemap file.
If you do not find a sitemap on your website, you can create one yourself. There are lots of tools to help with this, including XML Sitemap generator which is free for up to 500 pages, but you will need to manually remove any pages you don’t want to be included.
Step 2: Locate Your Robots.txt File
You can check whether your website has a robots.txt file by typing /robots.txt after your domain) for example, https://befound.pt/robots.txt.
If you do not have a robots.txt file then you will have to create one and add it to the root directory of your web server. To do this, you will need access to your web server. Usually, it is put in the same place where your site’s main “index.html” lies. The location of these files depends on the kind of web server software you have. You should consider getting the help of a web developer if you are not well accustomed to these files.
Just remember to use all lowercase for the file name that contains your robots.txt content. Do not use Robots.TXT or Robots.Txt as your filename.
Step 3: Add Sitemap Location To Robots.txt File
Now, open up robots.txt at the root of your site. Again, you need access to your web server to do so. So, ask a web developer or your hosting company for directions if you don’t know how to locate and edit your website’s robots.txt file.
To facilitate auto-discovery of your sitemap file through your robots.txt, all you have to do is place a directive with the URL in your robots.txt, as shown in the sample below:
Sitemap: http://befound.pt/sitemap.xml
So, the robots.txt file looks like this:
Sitemap: http://befound.pt/sitemap.xml User-agent:Disallow:
NOTE: The directive containing the sitemap location can be placed anywhere in the robots.txt file. It is independent of the user-agent line, so it does not matter where it is placed.
You can see this looks in action on a live site by visiting your favorite website, and adding /robots.txt to the end of the domain. For example, https://befound.pt/robots.txt.
What If You Have Multiple Sitemaps?
Based on Google & Bing’s sitemap guidelines, XML sitemaps shouldn’t contain more than 50,000 URLs and should be no larger than 50Mb when uncompressed. So in the case of a larger site with many URLs, you can create multiple sitemap files.
You must list all sitemap file locations in a sitemap index file. The XML format of the sitemap index file is similar to the sitemap file, making it a sitemap of sitemaps.
When you have multiple sitemaps, you can either specify your sitemap index file URL in your robots.txt file as shown in the example below:
Sitemap: http://befound.pt/sitemap_index.xml
Or, you can specify individual URLs for each of your sitemap files, as shown in the example below:
Sitemap:http://befound.pt/sitemap_pages.xml
Sitemap:http://befound.pt/sitemap_posts.xml
Hopefully, you’re now clear on how to create a robots.txt file with a sitemap location. Do it, it will help your website!
Have you located your sitemap in your robots.txt file yet?
Why Sitemaps Auto Discovery Matters
Back in 2006, major search engines (Yahoo, Microsoft, and Google) supported submitting XML sitemaps via standardized protocols. Initially, you had to submit sitemaps through their respective webmaster tools. However, in April 2007, they introduced Sitemaps Autodiscovery via robots.txt. This means that search engines automatically check for sitemaps listed in your robots.txt file.
Conclusion
By adding your sitemap reference to the robots.txt file, you create a clear path for search engine bots. They’ll efficiently crawl and index your pages, ultimately improving your site’s visibility and SEO performance. Remember, a well-structured robots.txt and an XML sitemap are essential components of a search-friendly website.
FAQs
Can I have multiple sitemaps for different sections of my site? Absolutely! List each sitemap in your robots.txt file to cover all relevant pages.
What if my sitemap URLs change? Update your robots.txt file whenever you modify your sitemap URLs.
Do all search engines follow Sitemaps Autodiscovery? While major engines do, it’s a good practice to include sitemap references explicitly.
