Why Are All the Models Failing?

WHEN TIMES ARE TOUGH and response rates are falling, it’s not uncommon to hear managers complain that their models aren’t working.

More often than not, it’s a misunderstanding about what it is that response models do and don’t do.

In most situations where this problem arises, what are called response models are logistic regression models that were never intended to predict the absolute level of response for promotions of different depth. (By depth I mean how deeply you go into your prospect or customer files.)

But if regression models don’t predict response, what do they predict? They don’t really predict anything. What they do is spread an average, and if they’re really good models, their ability to spread an average will hold up well over time.

Most response models are based on one or more promotions that took place in the past, and for which results are known. To keep it simple, think about one mailing. Let’s assume the mailing delivered a 2% response. What a good model will do is allow you to predict the expected response rate for each individual promoted. The average of all the expected response rates should be 2%.

Let’s assume time has passed and you’re ready to do the next promotion. And let’s further assume we want to promote to only the top 70% of the file, and that the file has been updated and rescored. A reasonable but incorrect assumption is that if the model works, the cumulative response rate will be 2.4%. But that would only be true if there were no changes in the environment, and that promoting the entire file would have again resulted in a 2% response rate.

But what if conditions had changed (due to fatigue, seasonality or competition), so that mailing to the entire file would have resulted in only a 1% response rate? The reasonable assumption would be mailing to the top 70% of the file would result in a cumulative response rate of only 1.2%.

The critical point is that logistic regression response models don’t tell the whole story. To accurately determine what’s going to happen if you promote to only a fraction of your target, you must first predict what will happen if you promoted to the entire population. Then you can use your regression models to spread the average expected response. If nothing else has radically changed, your predictions should be valid.

David Shepard is president of David Shepard Associates Inc., Dix Hills, NY.