All Search Engines Love Spiders - How Meta Commands Can Help You Love Them Too
Nearly all search engines utilize spiders (which are also known by their original name, robots) to go out and scour the web looking for web pages. These search engine spiders then bring the data back to be indexed by the engine.
Since roughly 1996, individual meta commands have existed that can be used on individual web pages to modify how these search engine spiders behave. The most useful of these commands are fairly universal and respected by almost all search engines. What follows is a list of some of the more popular spider commands and instances in which you might want to use them.
meta name=”robots” content=”index”
This meta command is one of the most common ones used -
...posts. It's no wonder there is so much information listed, everyone wants to put their two cents worth. You should too. It's like one huge list of all the Web Sites in the world put into a large file. But ...
meta name=”robots” content=”follow”
The follow command is different from the index command. It basically requests that the search engine spiders follow the links that are on a particular page. Again, however, this piece of code is completely unnecessary because all search engines are going to follow the links on
...button. You have to please the robots and the humans. If you overdo SEO you won t please the humans. You could also tick off the robots and get blacklisted. If you try to be an underachiever and ignore SEO, ...
meta name=”robots” content=”noindex”
The noindex command, the opposite of the index command, tells search engine spiders not to index the content of a page. It’s important to note however that search engine spiders will still follow the links on a page that uses only this command.
When not used for legitimate purposes, this tag can be dangerous because it can put you at risk for penalization by most, if not all search engines. This is because you can use a noindex tag to hide pages with multiple links that you don’t want visitors to see but that you do want all search engines to index.
There are however some legitimate uses for the noindex command.
...the popular sites. In this case, proper implementation of the search engine marketing tips may be very effective. It will make your website presence felt in this crowded world of websites. Then only, everyone in the World Wide Web will ...
As Google mentions in its Webmaster Help Center:
“Consider creating static copies of [http://www.google.com/support/webmasters/bin/answer.py?answer=34431 ]dynamic pages. Although the Google index includes dynamic pages, they comprise a small portion of our index. If you suspect that your dynamically generated pages (such as URLs containing question marks) are causing problems for our crawler, you might create static copies of these pages.”
In cases like these, it is acceptable to use the “no index” command on the dynamic version of the page,
...typeface that need to be present in the system in order for it to work. This is the area where the most confusion reigns and attention must be paid. Each font is represented by a Screen font and a Printer ...
meta name=”robots” content=”nofollow”
This tag tells search engine spiders that it’s OK to go ahead and index a page and list it but that they shouldn’t follow any of the links that are on the page. This can be useful if, for example, you had some partners that requested a link on your site that you felt obligated to give, but you wanted to hold onto as much Page Rank as possible. Now this is of course between you and your own personal god, but you would be able to in effect have a partners page, add the nofollow
...publications around the world along with your biography. The effect is that (a) you become recognized as someone that has something to say about topics of interest in your industry, and (b) the backlinks to your site will increase traffic ...
meta name=”robots” content=”noindex,nofollow”
Obviously, noindex and nofollow are powerful tags - and in combination, they can make a page and the subsequent pages to which it links invisible to nearly all search engines. This combination command tells search engine spiders, “Do not read this page; do not follow any of the links on this page; do not include this page in your index.”
This command has its beneficial uses. For example, it can be placed on pages
...technique allows you to sit back and relax and wait for the customers to come in. 4. Blog marketing. This is a great avenue where your can post product reviews, recommendations, and testimonials of your most satisfied clients to better ...
meta name=”robots” content=”noarchive”
Finally, almost all search engines today, including Google and Yahoo, offer a cached version of a site alongside its listing that provides a snapshot of what the page used to look like. The noarchive tag, therefore, is available to be used in circumstances
...involvement, the combined efforts led to the 'tipping point' - an article written by Abbey Klaassen of AdAge Magazine... Clearly the way to be successful with a viral marketing campaign is tailoring your appeal to the right audience. Just as ...
For example, a business might run a one-time special that has a ridiculously low price to drum up some business while things are slow. The business will want to be able to shut that sale down as soon as sales are back up to a solid level. However, it is conceivable that someone could click on the cached version of the business’s site, see the old deal that was out there, and insist on getting it for themselves. By using the noarchive tag, you are telling
...or your products on the internet. Depending on what your marketing strategies are, you can choose methods of website promotion. Of course, when you use SEO practice means of getting your audience to your website, you'll enjoy lots of advantages. ...
Conclusion
The commands discussed above are just a few of the ones in existence, and new ones are being added frequently. While nearly all search engines support these commands, there are still some that don’t. The ones in this article, however, are fairly universally understood by search engine spiders, no matter from where they originate. As more universal commands are introduced, I will write about them in future articles.
(C) Medium Blue 2007
Scott Buresh is the founder of Medium Blue, a [http://www.mediumblue.com/ ]search engine optimization company. His
...Let's face it: Some websites focus primarily on top search results, some focus primarily on design, and others focus mainly on delivering quality products and services and a top notch customer experience. If your website is lacking in one of ...