Data scientist Dean Abbott has been focusing on data mining and predictive analytics for more than 25 years, and has authored and co-authored several books including “Applied Predictive Analytics,” “IBM SPSS Modeler Cookbook” and contributed a biographical chapter to “Journeys to Data Mining: Experiences from 15 Renowned Researchers.” He has also presented at numerous analytics conferences including Predictive Analytics World, INFORMS and Gateways Analytic Network and is Smarter Remarketer’s chief data scientist.
We sat down with Abbott to chat about the evolution of big data and predictive analytics, how technology has evolved to bring the benefits to more marketers and where the space is heading in the future. We also dig into how analytics and big data is helping retailers reach customers in ways they haven’t been able to before.
Chief Marketer: How has the predictive analytics space evolved over the past few years?
Dean Abbott: From a technology standpoint, it’s the same thing, but seeing the industry grow up 
around me has been great fun and I’m really excited about the direction of retail as one channel, but there are lots of industries that are using predictive analytics now that didn’t used to. I’m excited about where we’re at from a technology standpoint.
There has been a natural progression in thinking about how to use data that has taken place. Ten years ago, analytics was more about reporting, but it’s more about the business intelligence layer on top of the data. We’re trying to understand what people are doing, how many visits, how long they spend on a page, just trying to get a sense of what they are doing. That’s a natural first step, but predicting what the customer’s behavior is going to be near-term is the next step and how you make better decisions. The customer has done ‘X,’ what do we expect them to do and can we do something to change their behavior to be something more favorable for us? If they’ve been browsing around on a website and not buying, is there something that could be done for that kind of customer to change their mind to purchase something from the site? That’s where predictive analytics is going—trying to make that connection between historic behavior and what people will be doing in the near or long-term future.
CM: Has analytics technology evolved greatly over the past few years, or are marketers just now figuring out the best ways to leverage it?
DA: From a scientific, mathematics or statistics standpoint, there’s been very little change. Fundamentally, I’ve been teaching and doing the same things for 20 years. Regression is regression, and that’s been around for 100 years. It’s still the biggest winning algorithm that people use across the board.
The biggest change is in the data that’s being collected. Fifteen or 20 years ago, only the high-end, larger retailers would be collecting the data you’d need to understand what customer behavior was all about. Others were starting to collect data, but they didn’t collect the right kind of data in order to make decisions. Now that people are collecting data in real time, transactional, multiple sources of data, that’s really what’s changed. It’s become cheap enough to collect and store that.
CM: Are there any ways retailers are leveraging data analytics that they haven’t in the past?
DA: In the past retailers collected data, but it was all siloed. You have the in-store data, transactions, loyalty programs all in one data store. Then online is a different data store, with interactions as they visit stored. So there are different kinds of data stored differently, and different groups within an organization use that information separately.
The trend that is starting to emerge with leading-edge companies is integrating all of this data together. The buzzword is the omni channel view of customers, so the customer doesn’t operate in a vacuum. I may be the kind of a customer who browses online for research, but I only purchase in the store. If you’re looking at the silo, I look like a highly engaged but unconverted customer. But in reality, I’m a regular customer, and you could see that if you tied in the in-store behavior with the digital behavior. Looking at behavior on the web, mobile apps, mobile web, in-store and catalogs in isolation gives you only a partial view of what the customer is doing. It means you might not be providing the most relevant information to that person in the right location to help them better understand what they are most interested in.
CM: Where do you see predictive analytics in the retail space trending toward in the future?
DA: Predictive analytics has gotten people’s attention, the difficulty is that the retailers who are making the decisions are not statisticians, they’re not mathematicians, and predictive analytics can take a while to wrap your mind around. Unless you’re a big organization with dedicated teams to do this kind of predictive modeling, it can be a difficult hurdle.
The key is to make predictive analytics consumable for the retailer, so they don’t have to do it all, but they can leverage the advantage of analytics without having to build the models themselves. The second level is understanding what data to collect and how to integrate the data, which is another big hurdle that requires a multidisciplinary approach. They need to have teams in place who understand how to get the right data integrated together, so that you can make these decisions. The third part is fitting it all together.
You have big data and predictive analytics people trying to enter into the retail space, but many of them don’t have the domain expertise to understand how it should fit together, so you put together the wrong data and models that work well, but answer the wrong questions. It’s the marriage of the technology with the domain expertise that gives you the biggest bang for the buck. So educating mid- and upper-level management on what predictive analytics can and can’t do, and how you can leverage it efficiently is going to be a big hurdle that retailers have to overcome.
In the abstract, retailers want a better conversion rate, but that’s really not what they’re after. They’re looking at dollar-spend, mind share and wallet share, and conversion rate is one way to measure that, but it’s not the best way to measure that. You could build a model that maximizes conversion rate and you end up developing campaigns that end up targeting low-end, low-margin products. So you get elevated conversion rates, but it’s not improving the bottom line, because it’s not addressing the right metric.
Network
