In our four-part blog series, BIS is examining the phenomenon known as “the Digital Dark Age.”
In Part I, we defined and gave readers an overview of what the Digital Dark Age is and how our rapidly expanding world of information and creativity is driving us closer to this Digital Dark Age.
In Part II, we take a look at ways the Digital Dark Age could affect businesses (or even households) if they fail to properly prepare for ways to unlock and bring their dark data to life.
Today, in Part III, we talk about how to get at the Dark Data in your organization but perhaps more importantly, what to do with it once you you’re able to “grab” it.
There are ways to avoid “sinking” into the Digital Dark Age, but it requires proper planning and an understanding of the platforms that can help.
How To Get To Your Dark Data and What To Do With It
Technology has allowed your business to gather vast quantities of data on your customer and employee activities, both online and offline, and all as part of the rush to be included in the ‘Big Data’ craze.
Some of the information you gather is used for a specific purpose, but the rest is then simply forgotten about. Every business has a multi-terabyte tranche of digital data that nobody is paying attention to – or what is known as “dark data”.
This is gathered during normal business, but rarely, if ever, used for anything. Businesses aren’t always sure why they gather so much data – often just in case they might need it for legal reasons several months later.
Another form of dark data is the data you intentionally collect (say sales quotes), use it for one specific purpose (quote a data center redesign for Yourtown Bank), then store it, allow it to sit and take up space in a filing cabinet…and email server…or some other storage platform.
Statistics indicate that more than 90% of data collected by any given business falls into the category of dark data. You are paying to store this information, increasing your monthly outgoings with frightening regularity, even though you probably won’t use most of it ever again.
The information you’ve stored contains duplicate copies of information stored elsewhere, and all of it sitting in an unencrypted archive somewhere, in the hope that nobody ever finds it. The future goals of CIOs will be to secure the data you do need, while also managing the data you no longer need (or think you don’t need).
Delving into dark data can provide you with invaluable insights into aspects of your business you’d never paid attention to before…purchasing trends to predict future sales trends
with a very high degree of certainty
For most businesses the focus might be on how to eliminate most of the dark data you’re storing, but we suggest looking beyond that. Way beyond that somewhat myopic view of your data. There might be profit, or business opportunities, hidden in that mountain of archived data you just don’t know about. Think of it as drilling for oil, but your drill has to bore through millions of records (useless, old, unknown…) to find the really useful stuff. Then you have to be able to index, classify, normalize and yes, re-store all that information onto a platform you can access. What you find can be used to simply create an historical record of your business, but, to us, it’s far more valuable than that.
But where do you actually find this mysterious “dark data”. The answer is that most of it is right under your nose in the form of customer accounts, Internet and local network log files, shopping cart logs, emails, video files, survey data, financial statements, invoices, delivery notes or documents, employee records, text documents, spreadsheets, presentations, and even sensor data from car parking lots, entry and exit controls and environmental controllers.
Now, this vast swathe of data is stored across multiple platforms, and in a variety of formats. Your first challenge will be to analyze what dark data you have, aggregate it into a usable format, and then set about finding value in it.
Looking at the data can reveal a lot of important information, like whether or not employees are following correct processes when dealing with customers. Or, are work computers being used to access Facebook more often than the CRM system you’re paying your staff to use. You could check if sensitive emails have been sent outside your network, or even the behavior of suppliers when you’re ordering from, or paying, them.
In a real life use case, one of the country’s largest school districts decided to access decades worth of old water district bills across the many campuses that comprised the district. Access old water bills? What for? Your first reaction is…that’s just useless data. They then used their collection of newfound data and analyzed it against decades’ worth of credits they were to have received across their many campuses.
Statistics indicate that more than 90% of data collected by any given business falls into the category of dark data
What resulted was that the district was entitled to millions of dollars from the utility company. This is what we mean when we say think beyond superficial uses of your data. On the surface, it looked as though the school district just had piles of old, useless water bills. By using one of today’s modern data collection software platforms, they were able to ingest millions of bits of data, analyze it, and realize they were sitting on a stockpile of money owed to them. With today’s modern technology, there are ways to get at your stored data – even if it’s yellowing paper sitting in rusting filing cabinets that are just taking up floor space. You have the wherewithal to ingest that information, normalize it, index it, classify it…then make it usable and actionable to improve your business.
Delving into this type of data can provide you with invaluable insights into aspects of your business you’d never paid attention to before. You could, for example, look at previous purchasing trends, helping you predict future sales trends with a very high degree of certainty. This type of predictive ability would be of immense value in almost any industry, but especially in manufacturing and retail. Amazon is a great example of a company that mines dark data so effectively they almost seem to be psychic…And perhaps therein lies the key message of utilizing dark data – if you don’t do it your competitors most certainly will. He who has access to the data holds the keys to the kingdom.
In our final blog entry we’ll look at how our society can (and likely will) move out of the Digital Dark Age and into an Age of Enlightenment. After all, we only need analyze on our own human historical data…to predict…perhaps guide…our future destiny.
For a PDF version of this blog entry, click here.