Learn about web site design and development, affilite programs, SEO, search engines, Google, and more.

Google Crawl Rate

If you’ve submitted your website through Google’s Webmaster Tools you’ll see an option under the Tools Menu for “Set Crawl Rate”. The Crawl Rate determines how often Google’s robots (the Googlebot) will visit your website to spider your web pages and add them into it’s search engine index. The options are Slow, Normal and Fast. The higher the crawl rate the better because you want Google to find your new pages quickly and have them searchable by Internet surfers.

For most websites the Crawl Rate is automatically set at “Normal”. If the Normal Rate is a strain on your website’s servers you can change this to Slower to have your site crawled less.

Today when I looked at my Webmaster Tools, I had a yellow note that said I could select a Faster Crawl rate for my website if I wanted to.

google crawl rate

Of course I wanted to. Who wouldn’t want Google to crawl their website faster? I’ve heard that with the Faster Crawl Rate pages are added nearly instantly to Google’s index. After I selected the Faster Crawl Rate option another note said my site would be on a Fast Crawl rate for the next 90 days. Cool!

increase google crawl rate

So I guess we’ll see how it goes for the next 90 days.

how to get a faster google crawl rate
My current crawl rate is about 1,800 max pages per day. Here’s the current graph. I can’t wait to see what that will change to.

increase google spidering

As you can tell, I was very happy to see that I had this option available to me. It tells me that the Googlebot noticed that I’ve been working hard at providing unique content on my website and I’m being rewarded by having my pages crawled faster. I’m very excited about the possibility of my pages being indexed as soon as I upload them to the Internet. I’ve experienced this only one time before and that was with some Father’s Day Printable Greeting cards. Each time I added a new card to the page I saw that the googlebot picked up on it and indexed the new info immediately.

If you want to increase your websites crawl rate, you can add fresh and unique content to your website as often as possible. The Googlebot will notice it and reward you as well.



Content © 2008 Hoover Web Design http://www.hooverwebdesign.com

June 26th, 2008 at 7:44 pm

Comments are closed.