Training Google To Crawl Your New Website

How To Series

Getting Google to pick up your posts within 30 minutes is something you have to train Google to perform so it will crawl your site more frequently.

Getting your site to be picked up by search engines is a very important part of the process. There are so many different things which may have an impact as to how much traffic your site might get and it is critical to get anything that you post to show up in Google Search in a reasonable period of time. One might ask what is reasonable and for every person you might ask, you would get a different answer. Much depends on how often you plan on posting things and what you desire. What I want to see is Google showing my searches within 30 minutes of posting them. So, in a very unscientific method, I created a new web site to see what I could do to get Google to show my posts in that time period. This is a WordPress site which was not previously set up and I started this once the site was set up. This testing has nothing to do with traffic to the site, only that Google shows my posts in a reasonable period of time.

Google and Webmaster Tools
Google is the big search engine that everyone focuses on to try and get their sites to show up in the search results on page 1. How you do that is the subject of SEO related articles and you can find that elsewhere. The first thing which you must do to get Google to find your site and have it show up in searches is set things up with Google Webmaster tools. There are other explanations on how to do that and you can find that elsewhere. Once you have been able to set up your site with Webmaster tools and configured what was necessary, you then need to add your sitemap to it. You will need to have a sitemap generated for your site in XML and with WordPress, there are a number of plugins which do this. We often use Google XML Sitemaps. The name of the file is normally sitemap.xml.

Google will crawl your site periodically and that is part of the problem with a new site. It is going to take some time for Google to start to crawl your site on a regular basis. When you first add your sitemap.xml file to the Webmaster tools, Google is going to crawl your site for the first time. It is after that which you want to train Google to crawl your site more frequently. I use the word "train" as you are trying to convince Google and all their algorithms to look at you site on an increasing basis.

You will want to post things at least 3 times a week to start and at various times of the day. Do not post things at the same time as Google will then learn to crawl your site at the same time everyday once it gets to that point. The key is to have Google crawling your site with 30 minutes of posting something. If you are striving for high volume traffic, you want it to pick up posts within a few minutes of them being posted.

The Training Test
Google is always reworking things, but here is what I did for a period of 40 days.

• Started posting articles everyday and attempted to maintain a minimum of 300 words in the articles.


• Missed posts on days 11, 12, 14, 15, 17, 18, 21, 36 and 37. This indicates that you do not have to post every day, but to establish your site it is probably a good idea.


• Links to the articles on days 13, 16 and 24. The value of links to your posts is an important item in driving traffic to your site and Google is seeing it as having value.


• At day 10, Google was crawling articles anywhere from 3 to 7 days. (1 to 10 days)


• At day 20, Google was crawling articles anywhere from about 3 days. (11 to 20 days)


• At day 30, Google was crawling articles anywhere from 1 to 3 days (21 to 30 days)


• After day 30, Google was display articles at the 30 minute mark. This is the goal when this started.


• At day 40, Google was displaying articles on average at the 10 minute mark

This test was only about getting Google to display a posted article within 30 minutes of it being put up and I was able to attain that value by “training” the crawling of the web site. This was not about driving traffic to the site as the site was getting 3 to 5 page views per day which was probably me looking at the posts, so most of the traffic came from my location, though Google Analytics is showing some traffic from other locations.

So this tells us that you can “train” Google to crawl your site more often to establish the baseline for that period forward. I have to assume that most web sites start out pretty good for the first 10 posts and then drop off which would indicate the improvement in crawling after day 10. .

Your results will vary as this was not a scientific endeavor and not an exact test. Checking for pages to appear in the Google Search was not done in a strict manner and the results may be better than what was recorded. This should give you hope in training Google Search on a brand new web site. If you are doing this on an existing web site, it may take longer to get similar results especially if you already have Webmaster Tools set up on the site because Google has history in place for the site.

Good luck.