Shine a light on dark data, which is data your business collects but doesn’t fully exploit, by learning about common use cases and data integration strategies.
Nearly all big brands know the value of collecting, storing, and analyzing data, but big data outputs often mean only a fraction of data generated actually gets put to use. According to Forrester research, over 60% of all business data captured and stored is not used outside of compliance and record-keeping.
This dark data, defined by Gartner as “information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes” is rarely analyzed and can leave large gaps in organizational strategy.
When every interaction, transaction, and engagement gets captured, brands must prioritize what gets immediately utilized and what gets pushed to the wayside for safe keeping. This often means un- or semi- structured data – like that amassed from server log files, networking, call records, machine data, or free text customer feedback – is left to hang out in the archive “just in case.” Antiquated formats or legacy systems can also keep data in the dark because of their inability to connect to modern analytics tools or because the devices themselves have become obsolete.
So, why go searching in the dark? Many think making effective use of dark data will emerge as a critical competitive factor in the years to come. Brands that are able to account for this data and bring it out of the dark can uncover insights about their customer base and piece together connections that unearth bigger marketing returns. One study even predicted that organizations capable of turning all relevant data into actionable information will earn $430 billion in productivity gains over their peers by the year 2020.
Read on to learn more about how dark data can be used to yield insights and what approaches organizations can take to capture and analyze dark data more deliberately.
What Types of Insights Can Dark Data Yield?
Organizations can analyze dark data points to develop greater context and reveal deeper trends, patterns, and relationships that go unnoticed during normal business intelligence and analytics activities. Website visitor logs, for example, include IP addresses which can be used to geo-map customer access points and determine patterns in brand engagement on a local level.
One popular fashion retailer tapped into dark data to look more closely at customer churn factors. By analyzing granular stats on how and when customers dropped off, the brand could make predictions as to what motivated customers to leave. In turn, the brand could craft more targeted, relevant, and proactive re-engagement strategies for each lost customer as opposed to generic “We Miss You!” emails.
In the healthcare industry, experts hold hope for dark data from IoT devices to reveal insights for better marketing as well as improved health care. The Apple Watch, for example, can detect heart arrhythmias with 97% accuracy. When shared with the owner’s healthcare provider, this data could be used to create customizable healthcare plans. Yet, only 10% of this data is used.
In this way, dark data can be harnessed to empower sales goals as well as operational goals such as improving services, or a bit of both simultaneously.
How Can Organizations Better Collect and Use Dark Data?
To begin making the most of dark data, brands need to account for all unused data, giving it structure, and integrating it into existing data processes. To do so, they will need to identify additional tools that make dark data integration possible.
One method can be to create systems that lend structure to largely unstructured events. For instance, 84% of outbound social-style link sharing only occurs via email. That means these “dark social” shares, unlike those via social platforms, are not easily measured by brands and typically not incorporated into attribution systems used by platforms like social media and PPC.
To make email dark social sharing trackable, brands could simply add a “share via email” button to their article or use a URL-shortening service, both of which can be configured to track shares through custom URLs and generate valuable statistics.
Similarly, nonprofit organizations can create a dedicated community portal for donors and members to interact with one another. Allowing people to share their social media credentials can connect the members’ social accounts to the organization’s CRM, improving identity matching and offering deeper insights on demographics.
Many organizations can also enhance their ability to see and sift through dark data by updating legacy systems. For example, one major airline lagged others in real-time pricing abilities because its pricing technology was outdated and extremely opaque — a quintessential “black box” system. The system’s inability to display data more than two years old led to a crisis one Christmas when the holiday fell on a Sunday and the system had no data to offer about past Sunday Christmases. By switching to a transparent price-setting system, the airline was able to integrate data across silos and access granular data for deeper analysis and improved price setting, turning dark data into direct ROI potential.
What’s Next in Dark Data Analysis?
Artificial intelligence has become a critical tool within dark data analysis since it can automatically mine data to add tags, structure, meta-information and more. Some brands now use advanced AI tools to analyze video, for instance, to mine transcripts or turn visuals into a form of structured data. Others use natural language processing to analyze free text or pick up on subtle social mentions.
Tools like these comprise a major frontier in modern business, encouraging brands like Apple to invest $200 million in purchasing an AI analytics company that specializes in uncovering dark data.
Future AI tools will offer organizations even more data than they already have, making it imperative that they have thorough and complete strategies in place. That includes dark data analysis, which can lead to more efficiency, better customer relationships and, ultimately, higher returns.
Great article and spot on – firms must maximize the value of their most precious asset – their data. Getting access to all of a firms data, including their dark data, in real time will be critical to their ability to drive data-intensive applications such as machine learning and AI. Most firms don’t have the data fabric necessary to bring together the entirety of their data landscape in to a single, global space. Those that make the investments in their data infrastructure will certainly reap the rewards.