Use Site Checker to See a Web Server the Way a Search Engine Spider Does
You can find out how a competitor’s Web server looks to a search engine spider by running the Site Checker utility. This allows you to look at the actual process that displays the page, which is on the server level.
Generally, an SEO-friendly site should be free of server problems such as improper redirects (a command that detours you from one page to another that the search engine either can't follow or is confused by) and other obstacles that can stop a search spider in its tracks. When you run the Site Checker utility, it attempts to crawl the site the same way a search engine spider does and then spits out a report. If you use the free Site Checker that is part of the SEOToolSet on http://www.seotoolset.com, the report lists any indexing obstacles it encounters, such as improper redirects, robot disallows, cloaking, virtual IPs, block lists, and more. Even if a page’s content is perfect, a bad server can keep it from reaching its full potential in the search engine rankings.
You can use any Site Checker tool you have access to, but this example uses the Site Checker mentioned above:
Go to http://www.seotoolset.com/tools/free_tools.html.
Under the heading Site Checker, enter the URL of the site you want to check in the Web Page text box, and then click the Site Checker button.

The first page of the Site Checker report for a classic cars Web page
This example runs the Site Checker report for a classic cars site, as shown in the above figure.
In the report shown above, you can see that they have a Sitemap.xml file which serves to direct incoming bots. The more important item to notice, however, is the number 200 that displays in the Header Info section. This is the site’s server status code, and 200 means their server is A-okay and is able to properly return the page requested.
The chart in the following table explains the most common server status codes. These server statuses are standardized by the World Wide Web Consortium (W3C), so they mean the same thing to everyone. (The official definitions can be found on their site at www.w3.org/Protocols/rfc2616/rfc2616-sec10.html if you want to research further.) This table boils down the technical language into understandable English to show you what each server status code really means to you.
| Server Status Codes and What They Mean |
| Code |
Description |
Definition |
What It Means (If It’s on a Competitor’s Page) |
| 200 |
OK |
The Web page appears as expected. |
The server and Web page have the welcome mat out for the search
engine spiders (and users too). This is not-so-good news for you,
but it isn’t surprising either because this site ranks
well. |
| 301 |
Moved Permanently |
The Web page has been redirected permanently to another Web
page URL. |
When a search engine spider sees this status code, it simply
moves to the appropriate other page. |
| 302 |
Found (Moved Temporarily) |
The Web page has been moved temporarily to a different
URL. |
This status should raise a red flag. Although there are
supposedly legitimate uses for a 302 Redirect code, they can cause
serious problems with search engines and could even indicate
something malicious is going on. Spammers frequently use 302
Redirects. |
| 400 |
Bad Request |
The server could not understand the request because of bad
syntax. |
This could be caused by a typo in the URL. Whatever the cause,
it means the search engine spider is blocked from reaching the
content pages. |
| 401 |
Unauthorized |
The request requires user authentication. |
The server requires a login in order to enter the page
requested. |
| 403 |
Forbidden |
The server understood the request, but refuses to fulfill
it. |
Indicates a technical problem that would cause a roadblock for
a search engine spider. (This is all the better for you, although
it may only be temporary). |
| 404 |
Not Found |
The Web page is not available. |
You’ve seen this error code; it’s the Page Can Not
Be Displayed page that displays when a Web site is down or
nonexistent. Chances are that the Web page is down for maintenance
or having some sort of problem. |
| 500 and higher |
Miscellaneous Server Errors |
— |
The 500–505 status codes indicate that something’s
wrong with the server. |
The other thing you want to glean from the Site Checker report is whether the page is cloaked. The Cloak Check runs through the site identifying itself as five different services — Internet Explorer, Mozilla Firefox, Googlebot, Slurp, and MSNbot — to ensure that they all match.

Cloak Check information from the Site Checker report
To manually detect whether a competitor’s site uses cloaking (showing one version of a page’s content to users, but a different version to the spiders), you need to compare the spiderable version to the version that you are viewing as a user. So do a search that you know includes that Web page in the results set, click the Cached link under that URL when it appears. This shows you the Web page as it looked to the search engine the last time it was spidered. Keeping in mind that the current page may have been changed a little in the meantime, compare the two versions. If you see entirely different content, you’re probably looking at cloaking.

SEO Glossary
404 error logs
The server record of every time a page could not be displayed on a Web site.

SEO Glossary
A/B testing
A comparison between the old version of a Web site (A) and the new version (B) to see which one performs better.

SEO Glossary
absolute link
A link that contains the whole file path to a document, including the protocol used to get the document (e.g., http://), the server the document is found on (www.mysite.com), the directory it is located in (images), and the document name (vacation.jpg).

SEO Glossary
acquisition
The measure of users that a Web site attracts. Acquisition metrics come from Web analytics data and include percent of new visitors, average number of visits per visitor, and average pages viewed per visitor.

SEO Glossary
ad group
One or more ads that target a set of grouped keywords.

SEO Glossary
advanced search operators
Special terms that you can insert in your search engine query to find specific types of information that a general search can't provide.

SEO Glossary
algorithm
A mathematical formula a search engine uses to establish which Web pages are the most relevant to a user’s search query. Algorithms can be fairly simple or multi-layered and complex.

SEO Glossary
anchor text
The words that make up a hyperlink that a user clicks.

SEO Glossary
archive
A storage area on a server where older Web content is out of the way, but still accessible.

SEO Glossary
Astro-turfing
A term used for a fake grassroots market campaign; this derives its name from AstroTurf, which is artificial grass.

SEO Glossary
backlink
An incoming link to a Web page from another site.

SEO Glossary
bad neighborhood
A Web site that has been banished from a search engine index because they were caught spamming or using other sneaky methods to fool the search engines into giving them a better ranking.

SEO Glossary
bandwidth
The rate at which data moves from one point to another over an Internet connection.

SEO Glossary
behavioral search engine
A search engine that tries to guess what exactly a user is looking for based upon their previous search inquiries.

SEO Glossary
blacklist; IP blacklist
A list of sites suspected of illegal acts such as child pornography, e-mail spam, or hacking.

SEO Glossary
blended search
The integration of different content types onto a search results page, such as images, videos, news, blogs, books, maps, and so on.

SEO Glossary
blog
Short for ‘Web log.’ An online conversation medium used to cover a wide range of subjects, including entertainment, politics, fashion, lifestyles, and technology. A blog can be anything from a personal journal to a media and corporate platform for describing new products and services.

SEO Glossary
bounce rate
A measure of the percentage of people who leave a Web site right after entering a page on that site, usually within seconds and without visiting any other page on the site.

SEO Glossary
branding
Establishing a company name and associating it with that business; well-known examples include Nike and Xerox.

SEO Glossary
browser plug-in
A software application that enhances a Web browser’s existing features.

SEO Glossary
cache
The stored version of a Web page.

SEO Glossary
canonical site
In the case of Web sites with duplicate content, the site the Webmaster prefers visitors and search engines to access. The primary, main Web site.

SEO Glossary
CGI
An acronym for Common Gateway Interface.

SEO Glossary
chiclet
A Web icon or link that can be clicked to submit or vote for an article on a particular social news or bookmarking site.

SEO Glossary
click map
A report that overlays Web pages on a site and indicates, on a per-page basis, which links visitors are clicking on to go to other pages on the site. Often, the most clicked-on links are bigger, have a richer color, or a note indicating how many clicks the link received.

SEO Glossary
click-through rate CTR
The frequency (number of times) that an Internet ad is clicked on.

SEO Glossary
cloaking
A technique in which the content of a Web page presented to a search engine spider is different than that presented to a user’s browser; as a result, the spiders see one page, while the user sees something entirely different.

SEO Glossary
CMS
An acronym for Content Management System.

SEO Glossary
concatenate
Run words together without spaces.

SEO Glossary
Content Management System CMS
Software that helps you create, edit, and manage a Web site.

SEO Glossary
content siloing
A technique of organizing a Web site into subject themes by linking related pages together.

SEO Glossary
content stacking
Writing the HTML for a Web site in such a way that the page content is delivered to the search engine spiders before any scripts or navigation elements.

SEO Glossary
content syndication
Distribution of a Web site’s content to other sites.

SEO Glossary
conversion
An action taken by a Web site visitor that meets the sales or business goal of that site. This is also a term for site visitors who become customers.

SEO Glossary
conversion funnel
The path that a visitor to a Web site takes to perform a desired action, most commonly a purchase.

SEO Glossary
conversion metrics
Basic processes on a Web site that can be measured to determine whether visitors are performing desired actions, or abandoning the site.

SEO Glossary
conversion rate
The number of visitors who actually purchase something, sign up, or take whatever action is appropriate on a Web site.

SEO Glossary
cookie
A small text file that a Web server automatically assigns to a user's browser; cookies are used to track, authenticate, and maintain specific information about users.

SEO Glossary
cost-per-click
The amount an advertiser pays each time their Web ad is clicked on a search engine results page.

SEO Glossary
country-code top level domain ccTLD
The last part of an Internet domain name; specifically, the letters that follow the final dot of any domain name. A country-code TLD is specific to a particular country, such as .ca, .cn, .uk, or .mx.

SEO Glossary
CSS Cascading Style Sheet
A separate file that is used to control formatting of text and images on a Web site.

SEO Glossary
day-parting
A feature that allows a Web advertiser to specify when during the day their advertising is shown on search engine results pages.

SEO Glossary
deceptive redirection
A type of coded command that redirects a browser to a different location than what was expected via the Web link that the user clicked.

SEO Glossary
dedicated server
A Web server that is used exclusively by one Web site, and not shared with any other sites.

SEO Glossary
demographics
User data such as gender, age, and so on.

SEO Glossary
direct type-in traffic
Users who type a URL directly into their Web browser’s address bar.

SEO Glossary
directory
A list of Web sites a search engine can search through that’s typically compiled by people, rather than by computer programs.

SEO Glossary
distribution
The placement of a keyword throughout a Web page.

SEO Glossary
domain name
The base address of a Web site, such as mydomain.com.

SEO Glossary
domain name registrar
A company accredited and authorized to register Internet domain names.

SEO Glossary
doorway page
A Web page submitted to search engine spiders that has been designed to satisfy the specific algorithms for various search engines, but is not intended to be viewed by visitors. A doorway page is usually filled with text content that makes it rank high for a certain keyword.

SEO Glossary
duplicate content
Identical or similar content that appears elsewhere on a Web site or on the Web.

SEO Glossary
e-commerce site
Any Web site that sells a particular product or service.

SEO Glossary
elephant word
A big word that is so laborious to type and so obscure in usage that only a very serious searcher would think of entering it in a search engine query.

SEO Glossary
e-mail spam
Unsolicited e-mail that is sent indiscriminately to many people.

SEO Glossary
encrypted
Converted into a form that cannot be understood except by knowing one or more secret decryption keys.

SEO Glossary
engagement object
Any type of interactive media object on a Web site that engages the Web site visitors’ interest. Examples include images, videos, audio, games, and applications.

SEO Glossary
error code
A message that a Web server sends to a visiting browser when something goes wrong.

SEO Glossary
exit page
The last page a visitor is on before they leave a Web site.

SEO Glossary
frame
An HTML technique for combining multiple documents in different sub-windows, all within a single browser window.

SEO Glossary
geolocation
The identification of a Web page as belonging to or being relevant for a particular country.

SEO Glossary
geotargeting
A technique used by a search engine to personalize search results to include local listings for search terms, based on a computer’s IP address. This technique is often used when a person searches for items that involve brick-and-mortar businesses or services that need to be provided locally.

SEO Glossary
Golden Triangle
A visual heat map showing how people’s eyes scan a search results page and how long they look at a particular result before moving on.

SEO Glossary
guided search
A Web search where the search engine makes suggestions on queries; examples include Google or Yahoo! Suggests.

SEO Glossary
hack
Attempt to break into computer networks and bypass their security.

SEO Glossary
hosting provider
A third-party company that leases out Web space by the month or year, similar to office space.

SEO Glossary
.htaccess file
A control file that allows server configuration changes on a per-directory basis. The file controls that directory and all of the subdirectories contained within it. Usually, this file is placed in the root folder of a Web site.

SEO Glossary
Hyperlink, hypertext link
A link to another part of a Web site.

SEO Glossary
impressions
The number of times an ad appears in search results to search engine users.

SEO Glossary
incestuous links
Links to the properties of an individual or organization, or among a group of friends' sites, where the links are then passed off as legitimate links from outside sources.

SEO Glossary
index
The database of Web sites that a search engine pulls results from in response to search queries.

SEO Glossary
indexing
The process of taking raw data and categorizing it, removing duplicate information, and generally organizing it all into an accessible structure (think filing cabinet versus paper pile).

SEO Glossary
internal links
Links on a Web site going to other pages within that site.

SEO Glossary
internal site search engine
A search engine that searches only a particular Web site.

SEO Glossary
Internet backbone
The main hub connections of the Internet, which are primarily located in major cities around the world (Los Angeles, Denver, New York, and so on).

SEO Glossary
Internet forum
A message board where users can log on and post about topics on a related subject.

SEO Glossary
Internet Protocol IP address
The numeric code that identifies the logical address where a computer, server, or site resides on the Web.

SEO Glossary
ISP
An acronym for Internet Service Provider.

SEO Glossary
JavaScript
A scripting language used to add functionality to Web sites.

SEO Glossary
keyword
A word relating to Web site content that search engines use to determine whether the site is relevant.

SEO Glossary
keyword density
The measurement of the number of times a keyword or keyword phrase appears on a Web page, compared to the total number of words on the page.

SEO Glossary
keyword phrase
A search query containing two or more words that Web page content relates to.

SEO Glossary
keyword stuffing
Putting every word, not just relevant ones, into the content of a keyword tag; repeating keywords over and over again.

SEO Glossary
landing page
The Web page that a visitor first goes to when clicking on an ad in a search engine.

SEO Glossary
link bait
Content that is created for the purpose of attracting attention and links to a Web site. Link bait is an accelerated version of a link magnet.

SEO Glossary
link equity
The perceived-expertise value of all the inbound links pointing to a Web page.

SEO Glossary
link farm
A network of Web sites that hyperlink to all the other sites in the group. Link farms are built for the express purpose of driving up link popularity.

SEO Glossary
link magnets
Elements on a Web site that are built in such a way that people naturally want to link to them. People happen upon the site, find the link magnet, and decide that it’s relevant and worthy of a link, so they create a link to the content on that site.

SEO Glossary
link request
An e-mail or other contact with someone that asks for them to link to a Web site.

SEO Glossary
local search engine
A search engine specializing in Web sites that are tied to a limited physical area (also known as a geotargeted area).

SEO Glossary
Local Shared Object LSO
Similar to a cookie, an LSO is a text file that can be read only by the Web site creating it. Browsers and anti-spyware programs can't delete them, and most users don't know how.

SEO Glossary
Long Tail
A statistical concept that says that items that are in comparatively low demand can nonetheless add up to quite large volumes.

SEO Glossary
Long-Tail search queries
Keywords, or search queries, made up of several words or a phrase.

SEO Glossary
markup
Formatting and other types of HTML code, such as Font tags to define the font style.

SEO Glossary
metric
A quantitative measure of something, such as a process, rate, or amount.

SEO Glossary
mirror
A full copy of a Web page or site.

SEO Glossary
multi-page analyzer
A program that measures the keyword density of multiple Web pages.

SEO Glossary
multivariate testing
Testing small changes to a Web site, such as the change of a certain font, or a button instead of an arrow. Typically, this involves testing many small changes to the same page at once instead of two totally separate pages, as in A/B testing.

SEO Glossary
negative keyword list
A list that is made up of the keywords that an advertiser does not want their ads to show up for.

SEO Glossary
news aggregator
A software program that allows users to subscribe to and read RSS feeds.

SEO Glossary
non-targeted content
Generic text that isn’t customized for a Web site’s various subject themes and keywords.

SEO Glossary
null test
In A/B testing, a test that is run on two A pages (pages on which no changes have been made) in order to establish a baseline and make sure the traffic isn’t adversely affected.

SEO Glossary
online social networking
Web sites where people can meet and interact with one another; examples include MySpace and Facebook.

SEO Glossary
on-page factor
The HTML tags and the visible content on a Web page.

SEO Glossary
on-page optimization
The changing of the underlying code of a Web page for the purpose of search engine optimization.

SEO Glossary
open source
A term for a Web site that allows anyone to access and edit its content.

SEO Glossary
open-source software application
A computer program whose source code is available for free to the public.

SEO Glossary
organic search results
Web pages that the search engines find on their own using their spiders. These results are not paid for by advertisers.

SEO Glossary
parameters
Automatically generated URL characters that carry information to the receiving Web page about a user.

SEO Glossary
pathing
Reviewing the flow, page by page, that a user takes while visiting a Web site.

SEO Glossary
pay-per-click PPC ad
Paid advertising that appears in search results, for which advertisers pay a fee every time their ad is clicked.

SEO Glossary
persona
A profile that represents a target audience based on calculated averages of their buying processes and demographics.

SEO Glossary
PPC
An acronym for Pay Per Click.

SEO Glossary
printer-friendly page
A separate version of a Web page that is designed specifically for printing. A printer-friendly page contains the same content as a regular Web page, but without the heavy images and advertisements that require a lot of printer ink.

SEO Glossary
query strings
The parts of a URL that pass data to a Web page. Query strings aren’t readable because they contain symbols (such as ?, &, and +) as well as codes, session IDs, and so on.

SEO Glossary
reach
The degree to which a Web site reaches customers. Measures of reach include overall traffic volumes, number of visits, number of new visitors, ratio of new to returning visitors, and visitor geographic data. Reach metrics depend on information from various sources.

SEO Glossary
reciprocal links
An exchanging of links where two sites link to each other: site A to site B and site B to site A. This is essentially a barter and generally does not contribute much, if anything, to link equity, but it is not harmful.

SEO Glossary
redirect
An HTML command that automatically forwards incoming links to a different Web page.

SEO Glossary
relative link
A link that references a file located in a physical directory relative to the current page or the root of all directories, so it can simply start with the page name and leave off the http:// and domain information.

SEO Glossary
response metric
An element that visitors are responding to on a Web site, such as an image, a newsletter, or an e-mail.

SEO Glossary
response time
How fast a Web server is and how long it takes for users to load a page from that server.

SEO Glossary
retention
A measure of how many customers remain once they come to a Web site.

SEO Glossary
reverse DNS lookup
Looking at what Web server a user is coming from.

SEO Glossary
ROI
An acronym for Return On Investment.

SEO Glossary
scalability
The ability to expand Web server resources as needed.

SEO Glossary
Scalable Inman Flash Replacement sIFR
A powerful technology that enables a Web designer to use a wide variety of fonts on their Web pages without sacrificing search-engine friendliness or accessibility.

SEO Glossary
scraper
A person who sends a robot to a Web site to copy (or scrape) the entire site and then republish it as their own.

SEO Glossary
search box
The text box in a search engine page where users type their search queries, or whatever it is that they’re looking for.

SEO Glossary
search engine
A Web application designed to hunt for specific keywords and group them according to relevance.

SEO Glossary
search query
The word or phrase that a user types into the search box of a search engine.

SEO Glossary
search verticals
Links to specialized vertical search engines that narrow a search into a specific type of result, such as images or news. Clicking one of these links takes a user to a results page with only images or only news.

SEO Glossary
segmentation test
Testing the variables in incoming Web traffic, such as finding out the demographics of the incoming traffic by asking them to answer certain questions.

SEO Glossary
SEO
An acronym for Search Engine Optimization.

SEO Glossary
SERP
An acronym for Search Engine Results Page.

SEO Glossary
server
The software and hardware that runs a Web site.

SEO Glossary
server logs
Records that measure the amount of traffic that a Web site receives. A server automatically creates a server log of all the activity it performs during a given time period, be it hours, days, or minutes.

SEO Glossary
session
The current time period a user is active on a Web site after logging on.

SEO Glossary
siloing
The process of organizing a Web site’s content into distinct subject categories, in order to group related content. Each silo has its own landing page and supporting pages.

SEO Glossary
site map
A Web page containing links to the pages in that Web site, similar to a table of contents.

SEO Glossary
site navigability
Link structure for moving around a Web site.

SEO Glossary
slip
A measure of how quickly visitors leave a Web page.

SEO Glossary
social networking site
A Web site where people can meet and interact with one another.

SEO Glossary
social news site
A site where news stories and articles from anywhere on the Web can be voted on by users, and the importance of a story or article is determined by the audience rather than by the editors of the site or source. Examples include Digg, StumbleUpon, and Reddit.

SEO Glossary
source code
The plain HTML code used to create a Web page.

SEO Glossary
spam
Any tactic or Web page that is used to deceive a search engine into a false understanding of what the whole Web site is about or its importance.

SEO Glossary
spider
A small program that search engines use to read and rank Web sites. Spiders are also referred to as robots, bots, or crawlers.

SEO Glossary
spider trap
A situation where a spider gets caught in an endless loop and is forced to abandon the Web page because it has no other alternative.

SEO Glossary
stemming
Using multiple forms of a word (such as customize instead of customization).

SEO Glossary
stickiness
A measure of how long a user stays on a Web page.

SEO Glossary
stop words
Very common words such as the, a, to, if, who, and so forth, which serve to connect ideas in search phrases but don't add much in the way of meaning to the phrases’ content. Stop words are ignored by search engines.

SEO Glossary
subdomain
A dependent domain set up within a primary domain. For example, in http://events.classiccarcustomization.com/, events is the subdomain, .classiccarcustomization is the domain, and .com is the top-level domain.

SEO Glossary
targeted traffic
Web traffic that is interested in a site’s product or service, and is thus more likely to provide conversions to the site.

SEO Glossary
top navigation
The (usually attractive and styled) links found at the top of a Web page, such as the Home Page, About Us, Contact Us, and specific category links; these top links are also referred to as global navigation.

SEO Glossary
top-level domain TLD
The root of a Web site’s URL.

SEO Glossary
traffic
The number of visitors a Web site receives.

SEO Glossary
trolling
The act of deliberately being rude and offensive just to make people angry on blogs and other Web forums; trolling generally gets a user banned from the blog or site.

SEO Glossary
UGC
An acronym for User Generated Content.

SEO Glossary
uptime
The percentage of time a Web site is up and running, not including scheduled maintenance periods.

SEO Glossary
URL
An acronym for Uniform Resource Locator.

SEO Glossary
user-agent sniffing
A process that identifies spiders coming to a Web site, enabling the Webmaster to keep out bad spiders.

SEO Glossary
vanity domain
An easy-to-remember Web address used to market a specific product, person, or service.

SEO Glossary
vertical search engines
Search engines that restrict their search by industry, geographic area, or file type.

SEO Glossary
virtual silos
An organizational scheme in a Web site that connects related pages using links instead of moving related Web pages into new directories.

SEO Glossary
W3C-compliant
Following HTML standards set by the World Wide Web Consortium (W3C).

SEO Glossary
Web analytics
The study of visitor activity on an individual Web site.

SEO Glossary
Web history
The complete records of a user’s previous Google searches and the Web sites they’ve visited or bookmarked.

SEO Glossary
Web metrics
The measurement of what's happening on the Internet, such as the number and types of people online; the number of broadband versus dial-up connections, advertisers, and advertisements (shapes, sizes, level of annoyance); and all things related to the Internet as a whole.

SEO Glossary
Web ring
A collection of Web sites from around the Internet that join together through interlinking in a circular structure.

SEO Glossary
Web server
The combination of software and hardware that runs a Web site. A Web server receives each user request and serves back the requested pages to the user’s browser.

SEO Glossary
white list
A list of Web sites or agents (such as spiders) that are approved to enter a Web site.

SEO Glossary
wiki
A type of information site that contains all user-generated content, such as Wikipedia.

SEO Glossary
World Wide Web Consortium W3C
An international consortium where member organizations, a full-time staff, and the public work together to develop Web standards.

SEO Glossary
XML
An acronym for eXtensible Markup Language.

SEO Glossary
XML site map
A document designed specifically to be readable by a search engine.