Are Your Webpages Lost in the Deep Web?
December 13, 2010
If your website information is to be found by searchers on the most popular search engines, and especially Google, you need to ensure that it is indexed as quickly as possible and is sufficiently well optimised to get onto the first page of the organic search results.
It might surprise you to discover that the vast majority of pages are never indexed by search engines and are therefore totally invisible to us as searchers.
The thing is the majority of pages never get indexed at all because of some very basic mistakes ….
Most of the reasons are technical like having Java on the home page (Google can't get past it to the rest of your pages unless it is set up properly and, sadly, lots of webmasters get this one wrong).
But there are other reasons. A recent article in New Scientist indicates that for every page indexed there are approximately 500 pages that aren't indexed. The reasons are many. In some cases the information is something you really wouldn't want people to see …. like your bank balance and transactions. In other cases it is your purchase of a train or plane ticket or the purchase of online shopping and clearly you wouldn't want that seen be searchers.
However the majority of information is of a type you would want people to see. For example you may have an openly accessible of information like the prices of your products or services that you would love people to access via a search engine. The problem is that if it is in a table, that is loaded via a database, it is largely invisible to the search engines. Some authorities claim that only 16% of the "visible" web is indexed. Some claim it is much less. Even at 16% it means only about 0.3% of the total is indexed!
A solution is being developed. A search engine that can search "form fronted" databases is being developed by Juliana Freire at the University of Utah.
Called Deep Peep it looks as if it is now possible to retrieve up to 90% of the information stored on form fronted data bases.
To try Deep Peep in it's beta format click the link
How useful was this post?
Click on a star to rate it!
We are sorry that this post was not useful for you!
Let us improve this post!
Thanks for your feedback!