Dealing with Algorithmic Actions if You’ve Been Dropped from Google’s Index
What if you have no Manual Action message? What if you have gone through all the checks and are sure that your site has been penalized? You’re working blind, in effect. Even if you fix the problems there’s no reconsideration request available to you; that’s reserved for manual actions. All you can do is fix the problems and wait for Google to reindex your site and (perhaps) remove the penalty.
If you’ve been dropped from Google’s index, the first thing to consider is whether you have, in effect, told Google to drop your site. This has been known to happen on a number of occasions, and the mistake can be minor. A few mistyped characters in your robots.txt file, for instance, can tell Google that you don’t want to be indexed.
Start by checking your robots.txt file, in particular the Disallow lines. For instance, the following is fine:
User-agent: * Disallow: /includes/
On the other hand, the following removes your site from the search engines:
User-agent: * Disallow: /
A small, simple mistake, and you’re dead in the water!
Google’s webmaster account provides some nice tools that help you check your existing robots.txt file. First, there’s the Blocked Resources report. Click the Google Index menu link, then select Blocked Resources; this report will tell you whether anything on your site is being blocked by robots.txt.
There’s also the robots.txt tester. Click the Crawl menu link and then the robots.txt Tester link. This tool lets you enter a particular URL to see whether it’s blocked, and also enter the text of your robots.txt file and test it. You can even request that Google quickly re-crawl your robots.txt file if you’ve modified it.
Note that you can also block search engines using the robots meta tags in individual pages. If, for instance, you have the following:
<META NAME=“robots” CONTENT=“noindex”>
you’ve just told the search engines not to index that page.
These are the most common problems, although there are other, far less likely, possibilities, such as the server being unresponsive to search engines. (You would have that problem only if you have a particularly incompetent or malicious server administrator, but it definitely happens.) Your Google webmaster account can tell you if Google’s having problems accessing your site; select Crawl and then Crawl Errors.