For the average and even not-so-average Internet user, the ?deep Web? is a murky realm. The notion of a hidden, secret corner of the Internet conjures up images of underworld cabals and illicit transactions ? the sort of affairs that might take place in a James Bond movie. While such images are not altogether untrue, there is a great deal of data with the potential for business applications.
To understand the deep Web, it is important to have a basic understanding of how Internet search engines work. When we do a search on Google or Internet Explorer, we are accessing indexed pages related to that search. These pages are found through the use of ?crawlers,? which find information related to words and essentially ?crawl? along the World Wide Web, accumulating information related to these pages and creating the vast indexes that become our one-stop shop for information.
Crawlers find information based on unique algorithms developed by their respective search engines. The way the pages are ranked has to do with the specific algorithm used by the search engine being used, along with those sites for which businesses have engaged the payment schemes made available by the search engine, which serves to enhance visibility (think of Google?s Adsense).
Web browsers keep the index of pages and their respective hyper links available for those searching for specific information. The number of pages indexed by a search engine runs into the hundreds of millions, which is why you can go through the pages of the results of a search ad infinitum.
The deep Web lies in pages and sites that do not have the appropriate criteria for which the conventional Web crawler can find information. It is a vast ocean of Web pages that have not been catalogued, thus making it an untapped resource for businesses seeking to obtain obscure information and data that might give them a competitive advantage. For example, a company that develops synthetic materials may desire access to the latest research in chemical engineering that is only available on an academic or professional association?s Web site.
The lurid side of the deep Web, ominously referred to as the ?dark Web,? is the place where illegal and illicit transactions occur, from illegal arms sales to prostitution, money laundering and child pornography. It grants anonymity through the use of dedicated software. A human rights organization could use the dark Web to contact human traffickers and/or their victims in an effort to locate such operations and engage international law enforcement.
The deep Web is full of possibilities that are nearly as infinite as the data that occupy it. Surface Web engines are still seeking ways to fully exploit it. A handful of sites, such as DeepPeep and DeepwebTechnologies, give direct access, or provide software that gives access to this realm.????
