Google has unveiled a new tool that will let Web publishers and marketers make sure their pages get indexed in its search engine.
Called Google Sitemaps, the new program is intended to improve Google search results for end users by making sure that Web sites get spidered more often and more intensively. But it will also have the effect of giving Web site operators added control over how frequently their pages get indexed by Google search bots. Web marketers may benefit from the ability to make sure that pages with time-sensitive campaigns or offers get indexed in a timely fashion and are more likely to be included in Google’s natural search results.
In a post on Google’s blog site, engineering director Shiva Shivakumar said the Sitemaps program will work in combination with Google’s regular indexing operations “to further improve the coverage and freshness of our index.” That will lead in time to delivering better and more search results from Web sites, he added.
“We’re undertaking an experiment that will either fail miserably or succeed beyond our wildest dreams, in making the Web better for Webmasters and users alike,” Shivakumar said in his blog.
How often Google indexes a Web page and the search rank it allots depend in part on the freshness of the page’s content. Pages that change often tend to get crawled more frequently, but some Web pages have trouble attracting the attention of Google’s search bots, and some types of dynamic Web content rarely get crawled. Previously, Web operators could only submit their home pages to Google and hope the rest of the site got indexed. Sitemaps is an attempt to fill those gaps in the Google index.
To take part in the Sitemaps program, Web site operators will create and post on their servers a file describing which pages on the site should be indexed, how often those pages are updated and when they were last modified, and their relative importance to other pages on the site. Google will not charge for participating in the Sitemaps program.
The new tool may also help Google stay ahead in the “index race” to search more Web pages than its rivals. Last November, Google claimed to be able to search about 8 billion pages of the Web. MSN claims to search about 5 billion pages, while observers estimate yahoo searches about 4 billion. A report from the University of Iowa estimated that in January 2005, the indexable Web consisted of 11.5 billion pages.
