How to Access the Web’s Hidden Riches

by August 15, 2017
Credit: Pixabay user geralt

Credit: Pixabay user geralt

Information is the key to great reporting. You want data, insights and sources that inform your work and are also different from what readers see in every other story on a particular topic. Here are some ways to mine the gold.

Locating databases

Databases can give you product names, employee lists, important collections of company documents, sophisticated analyses in the area of business you cover and much more. But because they’re only available through the database engine, they probably won’t appear when you do a Google search. They’re considered part of the invisible web, or the areas that are available but not so easily found because most of the contents aren’t necessarily on regular display on a webpage.

So try searching for whatever company, concept, industry name or other item you want and add the word “database” to the search. For example, say that you ran “bank enforcement database” through a web search.

You’d find a page for the Office of the Comptroller of the Currency with a link called Search Enforcement Actions Database. There would also be the FDIC Enforcement Decisions and Orders. Or you might substitute “spreadsheet” for “database,” as people often keep lists of interesting information in spreadsheet format. Some other substitutions could be “listings,” “postings” or “collection.”

Finding a directory

Directories are similar in a way to databases, as they keep categories of information. However, they tend to do so differently, using taxonomies to create a structural hierarchy for related data. Find a good directory and it can let you, step by step, work through large masses of useful data. Search for “construction directories” on the web.

There are directories of contractors, architecture firms and suppliers. You’ll also find directories of equipment, UL standards, companies that specialize in cold storage construction—all within the first few pages.

There are also general web directories that try to categorize as much of the web as they can. It’s a different way of searching that can help you see how distinct concepts are tied together, which can further inform more traditional keyword searches. Check some directory suggestions from Wikipedia.

Searching for secrets

You’d be amazed at the number of documents marked “confidential” or “not for release” that are uploaded to the web. It may be someone with access is trying to act as a whistleblower. It might be a mistake that someone forget to remove. Or it may be formerly restricted information that is no longer considered important.

As an experiment, I tried searching on Google for “company confidential” and “do not release.”

I could have used just one of the phrases or included the name of a company I was following. But, it was a complete fishing expedition. What I found on the first page of results was the separation agreement for an online brokerage’s former COO, noting annual base pay, potential additional amount in cash bonus, a note that a rent allowance for an apartment residence would continue for a particular time, amount in cash to be tendered in lieu of registered stock units and that health insurance would be paid by the company under a COBRA plan for a set period of time.

None of these techniques are guaranteed to work in every circumstance, but they’re always worth trying. You might get lucky.