Subscribe in a reader
HOME
Sitemaps Autodiscovery via Robots.txt
Now you can universally submit your Google sitemap via Robots.txt to Google, Yahoo, Miscrosoft and Ask. Looks like all have been assimilated but in this case it is a VERY good thing!
The new open-format autodiscovery allows webmasters to specify the location of their Sitemaps within their robots.txt file, eliminating the need to submit sitemaps to each search engine separately.
Comprehensiveness and freshness are key initiatives for every search engine, and with autodiscovery of sitemaps, everyone wins:
Webmasters save time with the ability to universally submit their content to the search engines and benefit from reduced unnecessary traffic by the crawlers
Add a line to your robots.txt file like this:
Sitemap: http://www.seobuzzbox.com/sitemap.xml (example here)
Google posts more on What’s New
I wonder if we will now be seeing more traffic to our websites from search engines who are not Google because of the new robot.txt addition?
Similar Post:
- Sitemaps for Google News
- Remove yourself from the Wayback Machine
- Yahoo and MSN adopt Google Sitemaps
April 11th, 2007 at 4:28 pm
What I do not get is wheither you can add more than one sitemap in the robots.txt file? And why on earth would they want the full URL, and not just the local server location (with base in document_root)?
April 16th, 2007 at 3:05 am
I presume that since they require the full URL it would indicate that you can have multiple sitemaps on the one domain.
April 16th, 2007 at 7:33 am
That does not make any sense. Why do you need the full URL to the domain when the rest of the configuration (robots restrictions) use / relative URLs? And Ask!, Yahoo!, wLive, and Google all say you can only have one sitemap in the robots file, but that sitemap might point to other sitemaps.
April 16th, 2007 at 10:53 am
What is you use subdomains?
April 16th, 2007 at 11:18 am
Then you have to create a separate robots.txt file on the subdomains’ root. Per today search engines counts subdomains (even http://www. !!) as separate domains.
April 16th, 2007 at 11:39 am
That all makes sense but I want clarification from Google.
April 16th, 2007 at 11:54 am
And Ask!, Yahoo!, and Live Search (MSN)… They all use the same format for sitemap auto-discovery.
April 16th, 2007 at 1:03 pm
What are you some kind of bot Daniel? That is what this post was all about right? Your point?
I haven’t had time to look yet but where is it stated that it is ok to have multiple sitemaps and sub domain sitemaps listed in robots.txt? Of course it should be fine but this is not always the case.
April 16th, 2007 at 1:33 pm
You can only have one Sitemaps: URI in your robotos.txt. However that sitemap may point to other sitemaps on the same domain.
April 19th, 2007 at 8:45 am
Not doesn’t make sense to me.
There is very little logic behind pointing to other sitemaps within one main sitemap if the robot can read which path the main sitemap is located in via robots.txt.
Why can’t he read multiple locations via robots.txt? After all the robots.txt file directs the bots to files.
I can’t see how a sitemap.xml file can be so different from a standard html file that the robot would choose to ignore it if there was more than one sitemap.
February 6th, 2008 at 11:55 am
Indeed. All top search engines are using the same format for sitemap auto-discovery.