Big data lives in a world of its own. It is messy, unstructured and constantly changing. To make use of it, marketing professionals need to leverage the power of data integration tools and get that data to a point where it's ready for analysis.
This process, however, comes with its own challenges, so we have made a list of our top 5.
Challenge 1: Extracting data from all sources
Data is coming at you from multiple directions. The number of data sources you use is likely quite large already and it is going to grow even further in the future. Collecting, transforming and sending data to a designated location is therefore becoming increasingly complex.
Clearly, locating the data is not the problem: it is in the tools you use and there is plenty of it. The challenge is to manage it so that when it lands in your target destination, it is easy to use and derive insights from. IT professionals Dipthi Karnad and Kapil Tulsan from ComputerWeekly advise integrating individual datasets one at a time, in order to avoid errors early on in the proces.
“Suppose you need to merge data from three applications: a merchandise management system, a customer database, and a product database. Break it down into individual datasets such as customer information, sales data, financial data, etc., and merge them one at a time.”
While it may seem more efficient to do everything at once, the truth is that your integration tool may become overwhelmed by the amount of data coming in. That, in turn, can lead to compromised data quality and faulty business decisions made further down the line.
Challenge 2: Achieving great data quality
As soon as you have your data collected from its initial sources, the challenge is to make sure that it is all properly cleansed and harmonised, so you can move on to deriving insights. In other words, you need to ensure the highest possible quality of your data.
Like we've said before, quality data is complete, consistent, timely and up-to-date. The last two of those characteristics can be particularly challenging, especially since data first enters your BI tool as one big clunky mess. There is no real structure or order to it in this initial phase. What’s more, different sources push data at different intervals of time and in formats that often vary among each other. The risk of getting out of sync on all your data is not that small, after all.
You can bring your data’s quality up by utilizing a powerful ETL tool, so you can eliminate any possible inconsistencies:
"To improve data quality, make sure your extract and transform processes occur at the source level, and not at the target level. The load and display processes can then be completed at the target level."
Challenge 3: Integrating data at the deepest level possible
The end quality of your data is almost entirely dependent on how well you clean, transform and integrate it into your business processes. If there is even the slightest inconsistency in your initial data, they will seep through to every next move you make.
The experts from Computer Weekly stress the importance of drill-down reporting. For example, if you are doing marketing for a retail brand: the data that you are collecting can be sorted in a number of categories from product and department, to store, city, country and region, to name a few.
"The best way to integrate data is at the product level, because then the BI tool can automatically merge it with the higher levels as needed. Integrating data directly at the company level, for example, will open up challenges when generating product-level drill-down reports."
The more accurate the basis of your data is, the fewer errors you are bound to encounter and of course, the better the decisions you make in the end.
Challenge 4: Planning for scalability
Even if it’s not on the top of our minds all the time, big data and scalability go practically hand in hand. As data growth does not seem to be slowing down, being prepared for even larger amounts in the future is a must. Unfortunately, however, only few organisations realise that before they embrace a specific BI tool.
While many still prefer to have their data housed on-premise, there are more and more cloud-based options out there. Deciding on a tool involves an evaluation of a number of factors: speed, cost, scalability and security are among them.
If you’re looking for speed, tight security and control, an on-premise data warehouse may be the choice for you. On the other hand, security is as tight in cloud-based solutons too, where protocols are being updated on a regular basis and potential bugs are being fixed as soon as they are detected, but speed might sometimes be an issue.
The areas in which a cloud solution is clearly taking the lead are a much lower entry cost, since there are no investments in hardware and infrastructure, and higher possibilities for scaling. After all, the storage space you can have in the cloud is practically infinite.
Challenge 5: Finding the right data talent
While one option is to have your data activities outsourced, there seems to be growing consensus around the fact that companies need to hire more talent with strong data skills.
In a survey we conducted recently, more than 50% of respondents said they have various people responsible for the data integration process, and 47% have already created a dedicated position - ranging from Data Analyst to Head of Data Technology and Chief Data Officer (CDO).
According to Wired magazine, “the CDO ensures that strategic importance of data is properly maintained and managed throughout the organization. With access to detailed market information, customer data and predictive analytics, CDO is in perfect position to identify ways in which this data can be harnessed.”
It looks like data and tech talent will be forming the core of many of today’s companies. Scott Hagedorn, CEO of Omnicom's new media agency Hearts & Science, is one of the people to highlight the shift in which organisations trying to become more data-driven: “I see the roles changing. That includes the rise of the marketing technologist as being a fundamental person on the team.”
At the end of the day, data integration makes the work of marketing, IT and other professionals easier and more efficient. It does not, however, come without its challenges, so do keep those in mind when crafting your long-term data strategy.
