Meta Robots Tag 101: Blocking Spiders, Cached Pages & More

The meta robots tag was an open standard created over a decade ago and designed initially to allow page authors to prevent page indexing. Over the years, various search engines have added additional support to the tag.

Let me start off by saying that if you DO want your pages in search engines, then DO NOT use the tag. By default, the major search engines will index any page they find. Yes, there is a form of the meta robots tag you can use to explicitly tell search engines to index your pages. It looks like this:

<meta name=”robots” content=”index”>

There’s also a form you can use that adds the command “follow,” which tells the search engines to index your page and also follow any links they find on that page to other pages, which they can then index. It looks like this

<meta name=”robots” content=”index,follow”>

You do NOT need to use either form if you DO want your pages in the search engines. Without either form, they’ll naturally index your pages and follow your links. That’s what they do.

I always joke that putting these forms of the meta robots tag on your web pages is like putting a Post-It note on your chest that says “breathe.” Hey, if you forget to look at that note, you’ll still breathe. That’s what you do, by default. And that’s what the major search engines do. By default, they inhale web pages without you putting up a meta tag telling them to do so.

Now if you DO NOT want your pages in a search engine, then it’s time to perhaps break out the meta robots tag, if for some reason the robots.txt alternative isn’t suitable. Want to keep a particular page out? Then put this on that page:

<meta name=”robots” content=”noindex”>

See the “noindex” value? That tells the search engines that see this page not to include them in their listings. Remember — as I explained before — this will not prevent the pages from being spidered. That’s because search engines have to keep revisiting the page in order to see if the tag is removed. The tag only keeps the page out. Here’s my earlier chart on that topic.

System	Robots. txt	Meta Robots	Yahoo Delete URL Option
Stops Crawling	Yes	No	No
Stops Index Inclusion	Yes	Yes	Yes
Stops Link Only Listing	No	No (Yes, for Google)	Yes
Why Use?	Easy to block many pages at once	Can’t access root domain	Don’t even want URL to appear or need page out fast

What if you don’t want links followed? Sure, you can do this:

<meta name=”robots” content=”noindex,nofollow”>

That extra command, “nofollow,” tells the search engines not to follow any links on that page. Google recently covered this more as an option. But as Google also explained, links from a page with this tag might still get crawled. That’s because if anyone else links to a particular page WITHOUT a nofollow value, then the search engine will follow that link.

So far, I’ve covered all the commands that were originally created with the tag back in May 1996. Since then, more commands (also called values or attributed) have been added. For example, Google writes today to summarize several options you can use. Quoting Google:

• NOINDEX – prevents the page from being included in the index.
• NOFOLLOW – prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
• NOARCHIVE – prevents a cached copy of this page from being available in the search results.
• NOSNIPPET – prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
• NOODP – blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.

At times, you may want to use more than one of these commands. I’ll get back to that. But first, how about another chart? I’ll cover the major commands you may want to use below:

COMMAND	Ask	Google	Microsoft	Yahoo
NOINDEX	Yes	Yes	Yes	Yes
NOFOLLOW	Yes	Yes	Yes	Yes
NOARCHIVE	Yes	Yes	Yes	Yes
NOODP	No	Yes	Yes	Yes
NOYDIR	No	No	No	Yes
NOSNIPPET	No	Yes	No	No
Robot Name	TEOMA	GOOGLEBOT	MSNBOT	SLURP
Does Robot Specific Tag Override All Robots Tag?	???	No	No	No

Several of these are already explained above, in what I quoted from Google. They work the same way for the other major search engines. I’ve also linked to help information from each search engine for more specific advice.

The NOYDIR command is fully explained in my previous Yahoo Provides NOYDIR Opt-Out Of Yahoo Directory Titles & Descriptions post. Only Yahoo supports this, but none of the other major search engines used Yahoo titles and descriptions for listings, so it doesn’t really matter for them.

Now on to the topic of a meta robots tag having multiple values. What if you wanted to keep a page from being cached by all the major search engines and also ensure that neither Open Directory or Yahoo Directory descriptions are used. First, you need the values of the commands to say this. From the table above, they are:

• NOARCHIVE
• NOODP
• NOYDIR

Next, you need to decide what robots to target. We’ll keep it simple for now. To target ALL robots, you use this value:

• ROBOTS

Now to the meta robots format. Without the values, it looks like this:

<meta name=”NAME-OF-ROBOTS-TO-TARGET” content=”COMMANDS”>

We replace that NAME-OF-ROBOTS-TO-TARGET part with the name of the robots we’re, well, targeting. As explained, that’s ROBOTS, in order to target them all. I’ll put it in bold below:

<meta name=”ROBOTS” content=”COMMANDS”>

Now we put in the commands we want to tell the robots, each separated by a command. The order doesn’t matter. Again, I’ll bold the commands:

<meta name=”ROBOTS” content=”NOARCHIVE,NOODP,NOYDIR“>

Voila! Put that tag ANYWHERE inside the header area of a web page like this:

<HEAD>
<meta name=”ROBOTS” content=”NOARCHIVE,NOODP,NOYDIR”>
</HEAD>

Then you will be telling all major search engines not to cache the page, nor to use Open Directory or Yahoo Directory titles or descriptions for you page listings.

Notice that in the tag above, there are no spaces between the commands. What if I did this?

<meta name=”ROBOTS” content=”NOARCHIVE, NOODP, NOYDIR”>

Google writes today that spaces make no difference. Use them if you want or not, the tag means the same thing. Microsoft tells me the same thing, as does Yahoo.

What if you did this, with no commas:

<meta name=”ROBOTS” content=”NOARCHIVE NOODP NOYDIR”>

Microsoft tells me this is fine. I didn’t ask Yahoo about this, and Google says commas MUST be used. So use commas and don’t be a pain.

Now what if you want to tell search engine different things. Maybe you want Microsoft not to use the ODP descriptions, Google not to cache pages, Yahoo not to follow links on a page and Ask not to index the page at all. Maybe you want to get your head examined for being so strange, too. But aside from your mental health, it is possible to do all this.

You need to have a robots tag for each particular search engine you want to target. See that chart above? At the bottom there’s a “Robot Name” row. That shows you the name of each search engine’s “robot” or “spider” that you’ll issue a command to. With the robot names, we then give each of them their specific commands:

<meta name=”TEOMA” content=”NOINDEX”>
<meta name=”GOOGLEBOT” content=”NOARCHIVE”>
<meta name=”MSNBOT” content=”NOODP”>
<meta name=”SLURP” content=”NOFOLLOW”>

You could also tell all robots to do one thing — say not to follow links — while also issuing a second robots-specific command such as telling only Google not to cache the page:

<meta name=”ROBOTS” content=”NOFOLLOW”>
<meta name=”GOOGLEBOT” content=”NOARCHIVE”>

But wouldn’t a search engine only follow the specific tag written for it? In other words, if you target Google with a specific command in the “GOOGLEBOT” tag, then wouldn’t it follow only that tag and ignore the other?

Google, Microsoft and Yahoo say they will honor them both. I don’t know about Ask. That’s why you see “???” in that “Does Robot Specific Tag Override All Robots Tag?” section of the chart above. I’ll try to get that answered.

What if you had more than one “all” robots tag like this:

<meta name=”ROBOTS” content=”NOFOLLOW”>
<meta name=”ROBOTS” content=”NOODP”>

As explained, you could easily do this instead:

<meta name=”ROBOTS” content=”NOFOLLOW,NOODP”>

But if for some reason you did do it the other way, Microsoft and Yahoo have told me that’s just fine. They honor the information in BOTH of the robots tags. Google’s post today says the same thing.

Finally, the Google post provides reassurance that capitalization doesn’t make a difference. I’ve shown things in various ways above, sometimes the commands in ALL CAPS, sometimes in lowercase. As Google says, case makes no difference. To quote their post:

Googlebot understands any combination of lowercase and uppercase. So each of these meta tags is interpreted in exactly the same way:

<meta name=”ROBOTS” content=”NOODP”>
<meta name=”robots” content=”noodp”>
<meta name=”Robots” content=”NoOdp”>

Ah, but what about something like this:

<MeTa nAMe=”RoBots” conTEnt=”NooDP”>

Well, Google didn’t go that far. But my experience over the past decade has been that meta tags are not case sensitive at all with the major search engines. So I think you’re safe in whatever case, for all the major search engines.

Read More

Elaboration of abbreviations

1.) GOOGLE - Global Organization
Of Oriented Group Language Of
Earth.
2.) YAHOO - Yet Another
Hierarchical Officious Oracle.
3.) WINDOW - Wide Interactive
Network
Development for Office work
Solution.
4.) COMPUTER - Common Oriented
Machine.Particu larly United and
used under Technical and
Educational Research.
5.) VIRUS - Vital Information
Resources Under Siege.
6.) UMTS - Universal Mobile
Telecommunicati ons System.
7.) AMOLED - Active-matrix organic
light-emitting diode.
8.) OLED - Organic light-emitting
diode.
9.) IMEI - International Mobile
Equipment Identity.
10.) ESN - Electronic Serial
Number.
11.) UPS - Uninterruptible power
supply.
12. HDMI - High-Definition
Multimedia Interface.
13.) VPN - Virtual private network.
14.) APN - Access Point Name.
15.) SIM - Subscriber Identity
Module.
16.) LED - Light emitting diode.
17.) DLNA - Digital Living Network
Alliance.
18.) RAM - Random access
memory.
19.) ROM - Read only memory.
20.) VGA - Video Graphics Array.
21.) QVGA - Quarter Video
Graphics
Array.
22.) WVGA - Wide video graphics
array.
23.) WXGA - Widescreen Extended
Graphics
Array.
24.) USB - Universal serial Bus.
25.) WLAN - Wireless Local Area
Network.
26.) PPI - Pixels Per Inch.
27.) LCD - Liquid Crystal Display.
28.) HSDPA - High speed down-
link packet
access.
29.) HSUPA - High-Speed Uplink
Packet Access.
30.) HSPA - High Speed Packet
Access.
31.) GPRS - General Packet Radio
Service.
32.) EDGE - Enhanced Data Rates
for Globa
Evolution.
33.) NFC - Near field
communication.
34.) OTG - On-the-go.
35.) S-LCD - Super Liquid Crystal
Display.
36.) O.S. - Operating system.
37.) SNS - Social network service.
38.) H.S - HOTSPOT.
39.) P.O.I - Point of interest.
40.) GPS - Global Positioning
System.
41.) DVD - Digital Video Disk.
42.) DTP - Desk top publishing.
43.) DNSE - Digital natural sound
engine.
44.) OVI - Ohio Video Intranet.
45.) CDMA - Code Division Multiple
Access.
46.) WCDMA - Wide-band Code
Division Multiple
Access.
47.) GSM - Global System for
Mobile
Communications.
48.) WI-FI - Wireless Fidelity.
49.) DIVX - Digital internet video
access.
50.) APK - Authenticated public
key.
51.) J2ME - Java 2 micro edition.
52.) SIS - Installation source.
53.) DELL - Digital electronic link
library.
54.) ACER - Acquisition
Collaboration
Experimentation Reflection.
55.) RSS - Really simple
syndication.
56.) TFT - Thin film transistor.
57.) AMR- Adaptive Multi-Rate.
58.) MPEG - moving pictures
experts group.
59.) IVRS - Interactive Voice
Response System.
60.) HP - Hewlett Packard.

Read More

Types of Websites

Categories for Types of Websites

In order to know what types of websites to build, it’s important to know the different types of websites there are. Websites are classified into various categories. Below is a list of the different types of websites based on the content and features offered. However, content can be varied, and with this in mind, we have tried to make segregation as intuitive as possible. Please note that these categories of websites are not distinct and can overlap, sometimes to a large degree; we call this fusion sites.

1. Search Engines & Directory Types of Websites(Google.com)

These are some of the most popular types of websites in the world. Some search engine companies, like Google or Gigablast, prefer to keep their homepage simple and stress on only one service, while Yahoo! and MSN have a more “portal” look which many services are advertised to the user like email, news etc. The portal provides a broader platform for the company to promote their different services but it also fragments the visitors. Classified directories and yellow pages websites list businesses (both online and offline) segregated into different categories or geographic locations. These may also include an internal search engine that helps users quickly locate information.

2. Informational Types of Websites (CNN.com)

These are the types of websites which are dedicated to the purpose of providing information – whether free or paid. These informational websites might be catering to a niche industry or be wider in their scope because of which they can be labelled as “news portals”. Information website can also contain updated time tables, TV guides, reference material, sports data (like cricket scores), weather and stock data.

3. Personal Websites (Stallman.org)

With tons of tools and services available, creating a website is no longer rocket science. The astronomical number of personal types of websites one finds on the web is mind-blowing! Personal websites and blogs are the fastest growing types of web site. Unfortunately, with personal websites, comes misinformation, poor designs and provides a sense of false-security for the general public in proper website design. Personal websites can contain any kind of information and are usually administered by a single individual who can choose whatever content they would like to put online. Anyone can now create a web site using online free hosting accounts that might also offer pre-made templates that are like “ready-to-go websites”, or with free and commercial versions of WYSIWYG HTML editors like Frontpage, DreamWeaver etc.

4. Blogs & Online Diaries (HuffingtonPost.com)

Blogs are like online diaries (the irony is, though you keep your hardcopy diary under lock-and-key, you want more and more visitors to read the online one). Blogs can be created by anyone using free services like Blogger.com (Google) or Windows Live Spaces (Microsoft). You can also install blog software to your pre-existing website and start putting your thoughts online in a matter of minutes! Because blogs are so easy to create, manage and even earn from (there are a lot of profitable blogs online, if you didn’t know), the web has increased manifold. And it’s all thanks to the many free blog services and a strong enthusiasm by the ever growing blogging community who keep maintaining and updating their web space regularly. From ramblings of a wandering mind to detailed and specific information on technical fields, the content of blog types of web site can vary widely.

5. Company Websites (NorthAmerican.com)

Company websites are by far the most important. Company websites range from the very small to the very large and can be a non-profit or for-profit business. Company websites have pages of information on the business, its services and clients. Company websites are often “static”. In other words, they don’t change much or very often. These are more like “online brochures” However, with modern Web 2.0, interactions with users and websites are becoming extremely popular and lots of companies are integrating new types of services and features into their sites. These sites are called “fusion” sites.

6. Forums (AngiesList.com)

Forum types of websites serve as platforms and promote interactions amongst the users. Unless specifically blocked and requiring a special invitation, you can join any forum on the web. Most forums cater to a specific industry or a field. So, for example, you will find forums on printers, banking and finance, biotechnology, even people with phobias of butterflies. Forums on websites, typically, contain categories that further narrow down the subject. Registered users can post questions or start a topic which other users can reply or add to. A lot of valuable information and knowledge exists in online forums and we urge you to join and actively participate in online discussions with people that share the same interests.

7. eCommerce i.e. Online Stores (eBay.com)

The web is primarily used for communication and information search, but many companies have set up shops online. In fact, some companies like the popular Amazon don’t have brick-and-mortar shops – the entire buying and selling takes place online. Online shops are not restricted to selling tangible products; they can also provide services and a good example of this would be travel and matrimony websites. Furthermore, you can integrate payment gateways, like Paypal.com and moneybookers.com, in your online shopping mall and process the orders and receive funds.

8. Web 2.0 Types of Websites (WordPress.com)

Web 2.0 is an amorphous concept and primarily encompasses websites that offer a particular service. The purpose of these websites can be varied – information storage, communication etc. Sites like Flickr.com, Google docs, del.icio.us fall into this category. Imagine it like this, browsers have upgrades such as the new Internet Explorer Version 9, and the World Wide Web is transitioning from static webpages into an interactive, highly-personalized experience. This is Web 2.0: the next “version” of the web.

9. Social Networking (Facebook.com)

The new mantra on the web is social networking. These websites can be classified as Web 2.0 sites but their sheer number and popular demands we have a separate category. Facebook, Orkut, LinkedIn, Twitter have been the rage past couple of years. From helping you locate schoolmates and past colleagues to microblogging, these websites have a whole array of utilities and features with more being added each day. Dating websites have grown over the years and now offer services quite similar to those of social networking (the idea is basically the same) hence they are a good fit in this category.

10. File-Sharing (RapidShare.com)

As internet speeds have increased and more and more people are using broadband connections, the importance of file sharing types of websites has grown. Megaupload.com and RapidShare.com are two prime examples of such types of websites. You can not only find images, but also music (sometimes even full albums) and video (entire movies). These types of websites offer both free and paid memberships – the former is usually a cut down version of the latter. However, because of the huge number of files stored on these sites, it is almost impossible to find the file you are looking for unless you know the direct link. Websites like loadingvault.com provides a search engine that helps you locate files based on your query. All of these are great Types of Websites that our Web Designers in Asheville can build for you. What’s your favorite Types of Websites?

Read More