Senior Content Analyst at Gartner
Dark data, dark data, dark data—repeat its name three times and it appears: a terrifying big data buzzword.
Do you know what’s most chilling about dark data? Most professionals don’t have the faintest idea what it is.
Yet these are hair-raising facts: 80 percent of business information is considered dark data, which means dark data is a problem that is stalking your business.
By 2020, as predicted in recent IBM research, 93 percent of all data will be dark data; business leaders need to address the massive competitive opportunity shrouded in its dark data, or else send their company to an early grave.
To unmask dark data and help your business face its fears about this daunting topic, here are answers to 7 questions about dark data you were afraid to ask:
Gartner defines dark data as the information assets businesses collect, process, and store during regular business hours and generally fail to use for other purposes.
But what does that exactly mean? Here’s what gives dark data its scary name:
Over the past decade, the volume of information businesses deal with has increased at a frightening rate. Organizations are gathering huge volumes of data from machines and customers—more than ever before.
Daily operations and infrastructure generate data that many companies are not armed to interpret. This challenge is only becoming harder. Consider this:
Information overload creates a need for big data, the collective practice of taking stock of all data an organization generates, making it visible, and then making sense of this insight to make better business decisions.
In all big data there is dark data—the unaddressed, untapped data assets that fail to get analyzed before falling off the grid. Even the best analytics tools and business intelligence (BI) strategies can’t identify and transform all big data into information that is understandable and useful to the business. Some of it falls through the cracks, most often hidden among people, networks, and machines. This is dark data, the extended, summer shadow of big data.
Most often, dark data is hidden or forgotten.
If we think of information as currency in today’s businesses—as critical a resource as physical, financial, and human investments—dark data is the lost change and pocket junk that absconds into the sofa; lost forever under the cushions.
Dark data is big data not living up to its full potential, which can be used to mine insights for knowledge that help improve and grow the business.
At the turn of the decade, Dell said 90 percent of data is estimated to be used once and never accessed again. IBM found similar results and reports that since 2015, 90 percent of data created over the past ten years was abandoned. Most of this data is ripe for analytics—juicy insights surrounded by an inedible peel.
Small business owner, Paul Bromen, of mattress review website UponaMattress.com notes a feeling of frustration when the value of dark data is realized, but progress is held back by business demands.
“Dark Data is a problem. Information collection is going up exponentially, but my ability [to deal with dark data] either goes up linearly or down as my growing business pulls my attentions in a thousand directions. I know there are million dollar insights sitting on those cloud servers, but I don’t have time to mine them.” Bromen said, pointing to an important dark data problem faced by small businesses.
The types of dark data present in your company will depend on your industry and the general types of information you collect or process.
Common dark data includes:
|Account information||Analytics reports/survey data|
|Biometric data||Customer call records/complaints/reviews|
|Emails and attachments||Geolocation data|
|Human resource information||Inactive databases, unused customer/system/machine information|
|Internet of things (IoT) diagnostics and status updates||Log files (e.g., servers, systems, architecture)|
Dark data sprawls across every level of a business. What’s disregarded and considered unnecessary by one individual or department may be valuable to another.
Should I be worried about the dark data in my company?
Yes — be worried.
If you don’t understand dark data and don’t take steps to identify it and ensure its proper governance you risk the following:
Security vulnerabilities – dark data provides no value and your business still incurs the attributed risks to host it. Dark data is less monitored—much of the time you don’t know its even there—making it a prime target for hackers, more susceptible to staff negligence exposure, and more difficult to discover when it’s breached.
Legal liabilities – dark data is still subject to government data regulations your company is responsible for. For example, the General data protection regulation (GDPR) enforces that companies are liable for their EU employee personal information regardless if the company is actively using this data or know it’s there. As any lawyer will tell you, “The dark data made me do it” is not a viable defense in court.
Lost competitive advantage – competitors are actively gathering insights from their dark data. Sitting on a gold mine of valuable data you can’t use does nothing. Timeliness of data is also increasingly a competitive factor, so dark data that sits for long periods of time atrophies and is no longer usable.
Dark data is not preventable.
Dark data is often welded to legacy systems, isolated in shadow IT, or scattered to various other unsanctioned places due to staff noncompliance or when authors of files move on from the organization.
Most dark data is sourced from unstructured data:
Unstructured information is more difficult to manage, automate, and therefore, more easily retracts into the dark recesses of the business.
As such, there will always be some amount of dark data in your company.
In fact, some data is better left dark. A common example of this is operational server data or the data output of automation technologies. This type of data is not easily interpretable or understood—save only to the most tech-fluent employees. Even then, these insights would be used for very specific, niche purposes. But it’s important to still protect even the most humdrum data.
The benefits of analyzing dark data are many. Here are some examples of how mining the business’ buried data can illuminate insights:
Dark networking data can be mined for insights. With analytics, you can analyze network security performance and gain insights into network activity patterns to check stress points and optimize resource use.
Records of customer-support conversations and other customer data—often underutilized and left to dissolve into the business data pool—are an opportunity for dark data analytics.
According to Cristian Rennella, CTO and CoFounder of elMejorTrato.com.ar, an insurance and credit review tool currently servicing the South American market, resurfaced customer dark data has proved extremely valuable:
“Thanks to a dark data initiative we were able to learn that depending on the time we answered the inquiries of our 21.5 million customers it would impact the probability of closing the sale by 85.2 percent.” said Rennella.
Rennella advocates for consistent effort to unearth and make the most of dark data.
“Dark data MUST be worked with business intelligence everyday and there MUST be a person in your company who is responsible for reaching insights that improve your processes.”
Until recently, dark data has not been a serious subject up for discussion in most enterprises, let alone smaller organizations with even less time to be stirring up sleeping data.
Often cited excuses include:
There’s abject horror in thinking about how you are going to explain this one to a boss or team who have never heard of dark data before. Start howling about the invisible “dark data” haunting your company—like you’ve seen a ghost—and you might be the one getting exorcised from the premises.
This conversation must happen.
To keep up with competitors, action is needed to figure out where dark data is, what dark data is, and how dark data is useful. If you do nothing, your unaccounted dark data will overload workers and processes, prevent efficiency, and shorten the lifespan of the business.
Dark data is invisible and you’ve likely forgotten it’s there. Therefore, the first step to manage dark data is to shine a light and identify it.
But to illuminate dark data, you need the right tools.
Data analytics solutions and the class of products known as business intelligence (BI) are helpful to unearth dark data deposits, increase visibility of information in the organization, and help melt down dark data into actionable advice and insights. Key benefits of this software include, data visualization, reporting, mining, and warehousing.
However, more than tools, action and a shift in strategy is needed:
1) Storing data “just in case” is no longer tenable. Data must be captured—the process by which data is brought into the fold of business databases and systems—with consideration for its purpose and expectations for its life while in your organization’s possession.
Data is not an indistinguishable zombie in the horde, but an individual. Data is an entity needing to be properly named, tracked, and curated.
2) Data management must be a top-of-mind practice. Information is more important than ever, and how well businesses manage their information assets is a way to competitively differentiate from rivals.
Small businesses are positively positioned to make these changes faster and more totally than larger organizations, and should use this advantage.
Tools, tips, and software catalogs to help your small business begin driving its dark data into the light:
Business Intelligence (BI) professionals share their success and failure stories with you.
Each app is scored using 5 factors, worth 20 points each, for a possible total of 100.