1.
Introduction
A sitemap is an important part for any website which is used by search engine optimization (SEO) which helps search engines like Google, Bing etc, to efficiently index the website's content. Blogger provides us with an automatic sitemap, but customizing your robots.txt file allows you to control how search engines crawl and index your blog. By configuring these settings, you can improve your site's visibility, prevent duplicate content issues, and optimize performance.
2.
Methods
2.1.
Method 1: Using Blogger’s Automatic Sitemap
Blogger automatically generates a basic sitemap that helps
search engines index your content. You can access it using these URLs:
●
For posts
sitemap: https://yourblog.blogspot.com/sitemap.xml
●
For pages
sitemap: https://yourblog.blogspot.com/sitemap-pages.xml
Replace yourblog.blogspot.com with your blog's actual URL.
2.2.
Method 2: Create a Custom Sitemap
If you want to generate a more comprehensive sitemap that
includes both posts and pages, follow these steps:
2.2.1.
Go to Blogger Dashboard
●
Sign in to your Blogger account.
●
Select the blog for which you want to create a
sitemap.
2.2.2.
Navigate to Settings
●
In the Blogger dashboard, click on Settings from the left sidebar.
2.2.3.
Enable Custom Robots.txt
●
Scroll down to the Crawlers and indexing section.
●
Enable Custom
robots.txt by switching the toggle on.
2.2.4.
Add a Custom Sitemap
●
Click on Custom
robots.txt and paste the following code:
|
User-agent: * Disallow: /search Allow: / Sitemap:
https://yourblog.blogspot.com/sitemap.xml |
Replace yourblog.blogspot.com
with your blog's URL.
2.2.5.
Save the Settings
This will ensure that both your
blog posts and pages are indexed by search engines, improving your website's
SEO.
2.3.
Advanced Custom Robots.txt Settings
To further optimize your Blogger
site, you can customize the robots.txt file with additional rules.
2.3.1.
Block Specific Pages from Indexing
If you don’t want certain pages
(e.g., example-page) to be indexed, add:
|
User-agent: * Disallow:
/p/example-page.html Allow: / |
2.3.2.
Block Search Pages from Indexing
Prevent Blogger's search pages
(?q=search-term) from being indexed to avoid duplicate content:
|
User-agent: * Disallow: /search Allow: / |
2.3.3.
Allow Googlebot but Block Other Bots
You can allow Googlebot while blocking others like Bingbot
or AhrefsBot:
|
User-agent: Googlebot Allow: / User-agent: AhrefsBot Disallow: / User-agent: Bingbot Disallow: / |
2.3.4.
Delay Crawling for Specific Bots
If you want to slow down how often bots crawl your site, use
Crawl-delay:
|
User-agent: * Crawl-delay: 10 |
(Note: Google
doesn’t follow Crawl-delay, but other search engines might.)
2.3.5.
Final Custom Robots.txt Example (All Rules
Combined)
Here’s a fully optimized version of robots.txt with all
settings applied:
|
User-agent: Googlebot Allow: / User-agent: AhrefsBot Disallow: / User-agent: Bingbot Disallow: / User-agent: * Disallow: /search Disallow:
/p/example-page.html Allow: / Crawl-delay: 10 Sitemap:
https://yourblog.blogspot.com/sitemap.xml Sitemap:
https://yourblog.blogspot.com/sitemap-pages.xml |
2.4.
Conclusion
By having the right settings for your Blogger sitemap and customizing the robots.txt file, you can improve search engine indexing, avoid duplicate content issues, and control how different bots access your site. So, implement these settings based on your SEO needs!
