Top
Reasons Why You May Not Be Indexed
1. INDEX TIME: First, make sure you've allowed enough time to
become indexed. The amount of time to allow is sometimes listed
on the search engine's submission page. Unfortunately, the engine's
own advertised times are often inaccurate or out of date. Average
index times often range from one to eight weeks depending on the
engine. Some engines like AltaVista and Inktomi offer paid options
if you wish to be indexed more quickly.
2. ALREADY INDEXED: Be sure you're not already indexed but
just don't know it. Unfortunately, none of the major engines
are kind enough to e-mail or notify you as to if and when you've
been indexed.
In addition, you cannot simply do a search on a keyword that
applies to your Web site and expect it to pop up at the top.
The method to determine if a page or domain has been indexed
varies from one engine to another, and in many cases, it's difficult
to tell for sure whether your pages are in fact in there. Never
assume that you're not indexed just because you searched for
a bunch of keywords and you never came up in the first few pages
of results. You could be there (i.e., indexed) but be buried
near the bottom.
3. MISSING PAGE: Make sure you have uploaded the pages to your
site before submitting them. This one will seem obvious to many
people, but submitting a page that does not exist or submitting
with a subtle typo in the URL is a goof we might all make at
one time or another. This is important since not all search
engines will notify you if the URL does not exist when you submit.
4. ROADMAP FROM HOME PAGE: Some engines have been known to
drop pages that cannot be traveled to from the home page. Think
of your Web site as a series of roads (i.e., links) from one
page to another. If there's no road from your home page to the
page you want indexed, a search engine may decide the page is
unimportant or of low-quality. You could submit the page directly,
but the engine may reject it or may drop it at a later date
when it finds no "road."
5. EXTERNAL LINKS: Some search engines such as Google have
been known to refuse to index Web sites that do not have any
other Web sites linking to them. Or, they may index your home
page but refuse to index any other pages until you achieve at
least one or more links from another domain. Or, they may index
you for awhile but then "prune" their database later
of all Web sites that did not achieve any external links within
a certain period of time. However, do not worry! You simply
need to establish some links and when that's done, resubmit
both your pages and the pages that link to you. Once you have
links to your Web site, it becomes much easier to get indexed,
stay indexed, and to achieve top rankings.
6. FRAMES: If you have content inside HTML frames, this can
cause problems with submissions. For example, the search engine
may index the main content of the page, but not the surrounding
menu frame. Visitors to your site will then find some information
but may not see the associated menu! It's generally best if
you can create non-framed versions of your pages. You should
then submit the non-frames versions of your pages, which can
of course link to your framed Web site. Alternatively, you can
enter your relevant text within the NOFRAMES area of a framed
page that most search engine spiders will read. However, don't
expect to achieve high rankings while optimizing the NOFRAMES
area. Optimizing a NON-framed page will often achieve better
results.
7. SPIDER BLOCKS: Search engine spiders cannot index sites
that require any kind of registration or password. A spider
cannot fill out a form of any kind. The same rule applies regarding
indexing of content from a searchable database. That's because
the spider cannot fill out a form to query that database. The
solution is to create static pages that the engines will be
able to find and index without performing a special action on
your site.
8. FREE SITES: Many engines no longer index pages from free
web sites or they limit the number of pages they will index
from these hosts. Sometimes they will get too many "junk"
submissions from free web site domains such as Geocities or
others. Therefore, some engines choose not to index anyone from
some of these domains. Or, more commonly, they limit the number
of pages they will accept.
9. GUILT THROUGH ASSOCIATION: If your Web site shares the same
IP address as many other Web sites on your host's Web server,
then you may find your IP quietly banned from something another
Web site on the same server did! It's always best to ask your
hosting service if your domain name has its own unique IP assigned
to it. If not, ask them to move it to its own IP to avoid the
potential of having your submissions ignored because of something
that a site sharing your IP did. We've heard from many people
who tried everything to be indexed only to find it was a snap
once they changed hosting services.
10. SUBMISSION LIMITS: Make sure you're submitting within the
recommended limits. Some engines do not like more than a certain
number of submissions for the same domain. If you exceed the
limit, you may find that all your submissions are ignored.
11. DYNAMIC PAGES: Dynamic pages are often ignored by the search
engine spiders. In fact, any URL containing special symbols
like a question mark (?) or an ampersand (&) will be ignored
by many engines. Pages generated on the fly from a database
often contain these symbols. In this situation, it's important
to generate "static" versions of each page you wish
to be indexed. In regard to the search engines, the simpler
the page is, the better. Does this mean, for example, having
a javascript to count visits to the page will prevent you from
being indexed, or lower your rankings? No. It simply means that
the search engine will most likely ignore the javascript and
index the remaining areas of the page. There is evidence that
going too far with fancy scripts and code on a page can hurt
your rankings if the bulk of your page consists of java or VB
scripts.
12. NON-INDEXABLE CONTENT: It's important to know the types
of content that the average search engine cannot index. Most
engines cannot index text that is embedded in images. Text that
appears in multimedia files (audio and video) will not be indexed.
Most engines cannot index information that is generated by Java
applets or in XML coding.
13. LARGE PAGES: If your site has a slow connection or the
pages are very complex and take a long time to load, it might
time out before the spider can index all the text. For the benefit
of your visitors and the search engines, limit your page size
to 50K or less. In fact, most Webmasters recommend that your
page size PLUS the size of all your images on the page should
not exceed 50K-70K total. If it does, many people on dial up
connections will leave before the page fully loads.
14. DEEP LINKS: If you submit just your home page, don't expect
a search engine to travel more than one or two links away from
the home page or from the page that you submitted. Over time
they may venture deeper into your site, but don't count on it
happening quickly. You'll often need to submit pages individually
that appear further down into your site or create more direct
links from the home page (either visible links or hidden links).
This way the search engines can find them. The technique of
submitting one page that then links to multiple other pages
you want found is called creating a "hallway page."
In many cases, this will not only get you indexed in cases where
they are ignoring you, but it will often improve your rankings.
That's because many engines assign "bonus points"
to pages they find on their own versus pages that were specifically
submitted to them.
15. UNRELIABLE HOSTS: If your Web site fails to respond when
the search engine spider pays a visit, you will not be indexed.
Even worse, if you are indexed and they pay a visit when your
site is down, you could be removed from their database! Consequently,
it pays to have a reliable hosting service that is up 99.5%
of the time. However, at some point a spider is going to hit
that other 0.5% and end up yanking your pages by mistake. Therefore,
it pays to keep a close eye on your listings and resubmit when
needed.
16. SPAM: If you have ever used any questionable techniques
that might be considered an overt attempt at spamming (i.e.,
excessive repetition of keywords, same color text as background...),
an engine may ignore or reject your submissions. If you're having
trouble getting indexed in the expected amount of time, make
sure your site is spam-free.
17. REDIRECTS: If your site contains redirects or meta refresh
tags these things can sometimes cause the engines to have trouble
indexing your site. Generally they will index the page that
it is redirecting TO, but if it thinks you are trying to "trick"
the engine by using "cloaking" or IP redirection technology
that it can detect, there is a chance that it may not index
the site at all.
18. PROPER DIRECTORY SUBMISSIONS: If you're submitting to a
directory site like Yahoo, Open Directory, Looksmart, or others,
then a human being will review your site. They must decide if
the site is of sufficient "quality" before they will
list it. I recommend you read the submission guide on the directory
tab of the WebPosition Submitter. It contains tips on how to
improve your chances of obtaining a good listing in these directories.
Getting listed in major directories first can help you get listed
elsewhere.
19. INDEX TIMES CAN FLUCTUATE: Sometimes engines will index
sites every 30 days fairly consistently and then suddenly stop
indexing most sites for several months. This can be frustrating,
but it does happen. Generally a major engine will not go more
than three to four months without refreshing its index. If you're
wondering if others are experiencing trouble getting indexed
on a particular engine, try asking around. You can often find
valuable real-world feedback on a search engine discussion forum.
20. PAGE LIMITS: If you have many pages indexed but are having
a hard time getting new ones recognized, be aware that there
are limits. Each search engine will only spider so many pages
of your Web site. This may range from a few dozen or three or
four hundred depending on the engine. Some people have even
been successful in getting far more pages indexed depending
on the engine. Google is one engine that tends to crawl deeper
into your site. However, how deep they go may depend on factors
like your link popularity. Sites with higher link popularity
are deemed "worthier" of more thorough indexing.
21. RANDOM ERRORS: Last but not least, sometimes the engines
just lose submissions at random through technical errors and
bugs. After all, they are managing a database of hundreds of
millions of pages. Therefore, some people like to resubmit once
or twice a month for good merit in case they do lose a submission.
Certainly if you've followed all the "rules" and are
still not listed, by all means, re-submit! Sometimes a little
persistence is all that's needed.
TIP: Once your page achieves a desirable ranking, it's best
not to continue submitting it. You risk the engine re-evaluating
the page and possibly reducing your ranking.
If any of the above scenarios apply to your submission, you
should take the appropriate actions and then re-submit. If that
still does not work, you should consider e-mailing or calling
the search engine and asking them politely why you have not
been indexed yet. Sometimes they will reply back with "Sorry,
there was a problem with our system and I've now made sure you'll
be indexed within the next couple days." Or, sometimes
they'll tell you why you were not indexed so you might correct
it. In other cases, they will ignore your e-mail and you'll
have to keep e-mailing or calling them until they respond. Still,
it's definitely worth the effort to get your site listed with
the major engines. This assumes you also take the time to optimize
your pages so you'll achieve top rankings.
|