Are you curious about the size of hidden web databases – otherwise known as the invisible web? In this article I’m going to outline not only what these areas of the web are all about, but also how you can use invisible web search engines to conduct some amazing research that goes far beyond simple Google research.
There are countless times when I’ve been digging through government and university websites to locate background information on a particular person or group, when I came across a document or a list of resources and wondered how and why no one ever came across them before.
The answer is that many researchers who use the Internet are purely “Googlers” – they use Google as their primary window into Internet content, and if it doesn’t show up in Google, then it doesn’t exist. The invisible web is the part of the Internet that remains out of reach of standard search engines.
How to Determine the Size of the Invisible Web
This is because search engines crawl through pages of HTML, and often the information that you’re looking for doesn’t exist in HTML form until you enter a database query into a form, and the script generates the HTML results on the fly.
By finding and searching through such databases, you can find data and information that no one else can find if they’re only using search engines. The size of the hidden web areas remain in question, but given the sheer growth of the Internet since the 1990s, I would guess that the invisible web contains a great deal of buried information just waiting to be discovered.
In this business, when you’re dealing with Intel types and spooks (or wanna-be spies), information is intelligence – and the more “intel” you have access to, the better positioned you are to outmaneuver those who seek to hide the truth from you.
The bulk of the “hidden” part of the Internet is what lies underneath those query forms. The way to gain access to those deep, hidden areas is by getting access to those database query forms.
Hidden Web Search Engines
Some of these include search engines that are programmed to dig more deeply into scholarly journals, articles and other academic libraries that typically get ignored by other search engines.
Google Scholar is the first place you should check out to search for scholarly journals and articles on the topic you’re hunting for. This is essentially Google’s version of an invisible web search engine that provides you with citations you can use to try and track down the original text.
Other excellent search engines that can help you uncover those query pages where you can dig deeper into the invisible part of the Internet include the following.
- CompletePlanet claims to access over 70,000 databases and unique search engines. The best option is to dig down through the subject topics on the main page.
- Digital Librarian is an amazing archive of hand selected resources, maintained by real life librarian Margaret Vail Anderson in Cortland, New York. These results aren’t computer guesses, they’re gathered by a very intelligent person.
- At Healia you can search through medical journals and clinical trials for information on diseases, symptoms and much more.
- Infomine is one of the best invisible web search engines that offers you direct access to numerous scholarly collections including Social Science and Humanities, Government info, Ejournals and more.
- Intute is similar to Infomine except it’s UK based – an excellent resource for UK government publications and UK academic publications.
- NewsCred is for researchers who hate search engine results that include so much garbage and so little legitimate news sites. NewsCred only returns search results from legitimate news sources.
- Re-Quest is an excellent directory search engine that offers hand-picked search results which, again, remove most of the garbage or useless information from the results.
- Scirus is one of my favorite search engines, where you can sift through not only scientific journals, but it also crawls scientist web pages, patents, courseware and more scholarly sources.
- WWW Virtual Library is a directory search site that looks like it’s stuck in the 90s, but it’s actually one of the best researched collection of Internet resources that you’ll find anywhere. This is because volunteers work on sections/subjects of the directory that they are experts in, and the results show.
Other Ideas for Dealing With the Size of the Hidden Web
As you start sifting through these directories and databases, you’re going to quickly realize how significant the invisible Internet is. These are vast areas of the Internet that remain completely unexplored, like the “New World” of the web – there’s “gold” in them thar’ hills!
All you have to know is how to make use of the resources that are available and start digging for the information you need. Google and Bing are great for getting started – but the real meaty part of the Internet comes along when you go beyond traditional web search and start using these non-traditional research techniques.
Do you have your own techniques to conduct research through the invisible web? Share your own experiences and resources in the comments section below.Originally published on TopSecretWriters.com