The Case for Simultaneous Concept Testing

Direct and interactive marketers have long understood the power and value of testing. Most tests, however, have focused on incremental elements of a communication, such as subject lines or envelope greetings. Marketers need to be more open to testing entire concepts simultaneously among significant numbers of customers, letting results determine which concept is most viable.

A lot of expensive consulting goes into program concepts and designs. But predicting the most effective design while in the design process is fraught with peril. As a marketer, you are trusting the instincts of your consultancy or employees, with little recourse once a program is in place. It comes down to sample bias. Internal employees are not a good source, since they have atypical knowledge of your company’s brand and other activities and have a built-in preference for your brand over other options. Consultancies (and internal employees, for that matter) rarely represent a cross-section that looks like your typical target audience.

What makes much more sense is to develop two, three, or up to five concepts, as an agency would, and test all of them with a substantial and representative sample of customers. And I don’t mean through an e-mail survey. There is too much evidence of inaccuracies between what people do and what they say they do. Put the programs in place, allowing customers to use their wallets to make real decisions about how they will respond. One program should emerge as the clear winner, vastly improving your chances for long-term positive ROI. Yes, your start-up costs will be higher, but the returns over time will far outweigh the initial expense.

What makes this a much more reasonable proposition now than it was just a few years ago is the dramatic improvement in segmentation and testing applications and service-based software models. Instead of requiring one-off development, programs can leverage multitenant software applications and set up segmented tests as part of the application itself. Under modern technology assumptions, the incremental cost of a second instance or program implementation is far, far less than it was when business rules had to be built into the solution. Even more important, if you plan for simultaneous tests before the project begins, the costs will be much lower than if you were to add a second concept halfway through a technology implementation.

What opened our eyes to this idea was an experience with an early client. This client insisted on running trials with multiple concepts. Since the incremental cost was trivial, we agreed to do the work to implement its ideas. In our client’s minds, there was an obvious choice for the program design, but it had a couple other ideas that it thought would work as well.

One concept quickly outperformed the other two. And it was not the one the client thought it would be. As familiar as the client was with its brand, a simpler, more straightforward program approach (in this case, a loyalty program) was the one its customers responded to. Determining causality post-test is always difficult, but it appeared that the simplicity of the concept improved usage and response rates. With all the messages clamoring for customers’ attention, this design and approach turned out to influence behavior most effectively, which we (and the client) would not have predicted.

Now we see clients testing as many as five concepts simultaneously. The incremental costs are low enough to allow ROI estimates to stay in range, especially given the smaller expected range of performance used as part of the business model—that is, we can use a lower standard deviation when calculating worst-case scenarios.

So the next time you are considering a new program of some sort, talk to your vendors about the costs for multiple concept tests. Negotiating this into your deal up front may cost much less than you expect and will pay substantial benefits down the road.

Michael Greenberg is vice president of marketing for Loyalty Lab, a San Francisco-based developer of customer loyalty programs for the retail industry, and writes a monthly column for CHIEF MARKETER.