IF YOU'VE BEEN IN DIRECT marketing for more than a few months, you already know the basics of setting up test cells that will deliver statistically projectable results measurement. However, you should consider creating an enterprisewide control group.
This type of control group differs from — and does not interfere in any way with — the A/B split testing typically employed to test and measure one creative or communications approach against another. Instead, it provides an ongoing measurement of the value of your company's global database marketing effort.
WHY DO IT?
Why would any company need such a measure? Well, after the first year, senior executives will want to know how the new program worked. Furthermore, two or three years down the road, that highly supportive CEO, CFO or CMO you rely on for funding might not be with the firm. The answer lies in proof — in the ability to demonstrate the overall value of your database marketing program to original supporters or, maybe, to a newly hired (and highly skeptical) brand-oriented replacement. That means real numbers, not anecdotes.
The basic idea is to isolate certain customers so that they do not receive any of the promotions or continuity communications that collectively make up your organization's database marketing program. Comparing those customers over time to the ones who do receive those promotions and communications will provide an incremental measurement of the effort's overall value.
This important tool can prove the validity of your decision to invest in database marketing technologies and processes. Perhaps even more important, it will provide trend measurement. As any database marketing program matures, there should be an increasing divergence in customer value between the control group and the rest of the customer universe. It will allow you to answer the question, “What's the company getting for all the money it's invested?”
The calculation for answering that question is fairly simple. Whatever measures are used, the average values in the control group should be compared with the same average values in the rest of the universe. Since the only variable is the presence of database marketing activity, the incremental difference between the two is the gain or loss that's a direct result of that activity.
BUT BE ADVISED…
A word of caution, however: Because the control group will be scaled for relatively gross measurements, you never should attempt to use it to evaluate more granular detail, such as creative testing, market-to-market comparisons and the like. The presence of the enterprisewide control group absolutely, positively does not eliminate the necessity for testing scenarios and setting up test cells to measure them properly.
The control group measurement, as well as the control group itself, should be a “background” methodology, one that goes on indefinitely and has zero effect on your other planning. Just learn to think of your marketable universe as being slightly smaller than it actually is. The best way to do this is to make the control group flag a global exclusion in your database processing, so there's no chance you'll ever contaminate the universe and thus reduce its viability as a statistically reliable measure.
Enterprisewide control group measurement should be limited to high-order data such as:
-
Dollar value of customers (revenue).
-
Number of relationships (cross selling).
-
Length of time on the books (tenure).
-
Retention or attrition (churn).
In all cases, the control group universe should be compared with the non-control group universe of customers and never broken down into any finer “slices” of data. This should be a monthly or quarterly report, with measurements both for the specific month the report was issued plus “rolling” 12-month averages.
CHOOSE AT RANDOM
Customers must be randomly selected for inclusion in the control group. In most cases, this can be done by simply flagging every “nth” customer in the file as a control group customer, then using the same nth method to add new customers to the control group with each update of the file. This will constantly refresh the control group to keep it homogeneous with the rest of the customer universe. Care should be taken, however, to make certain that the sample is random whatever method is used. Nth-ing the file might not be random if the incoming data is ordered in some fashion.
For a file of 2 million to 3 million customers I would recommend setting the sampling routine so that it will create and maintain a control group of at least 50,000 customers. For larger files, 1% of the universe or less usually will suffice. While this number probably will prove to be larger than required for statistical validity, it's best to err on the side of volume, at least in the beginning. Initial measures need to be “unassailable.” With experience, the control group's size might be reduced.
BY THE WAY…DON'T TELL SALES
And please note: It's very important that the methodology for establishing and maintaining the enterprisewide control group — and perhaps even knowledge of the existence of the control group itself — is not shared with the sales channel. If a salesperson knows who is in the control group and decides to take some special action to sell this previously untouched universe, the ability to deliver a true incremental measurement will be lost.
If you establish an enterprisewide control group, you'll always be able to answer questions about the value of what you do. It will provide useful, ongoing data that should help justify budgets when your company faces those inevitable crunch times.
Sooner or later, though, it probably will dawn on someone that your control group represents a lost opportunity cost to the company. The refrain usually goes something like, “Some of those customers you're ignoring are just like the ones who buy a lot from us. We need sales and you don't need to prove the value of database marketing. We get it! Let's unplug it!” If that happens and you can't sell the idea that you need a long-term measurement methodology, you need someone to intervene on your behalf.
People who follow this advice occasionally might experience one unintended consequence of the methodology: isolated adverse customer reaction. If you have two similar and very good customers who also happen to be very close friends and only one of them gets an offer that's creatively positioned as “a special benefit for our best customers,” that could come up in a conversation between the two. In such a case the control group customer who didn't get the right offer might feel snubbed and call to complain.
It doesn't occur often, but if it does, apologize profusely, explain that there must have been some mistake, extend the same offer to the caller, and, if possible, sweeten the offer with an extra discount or other benefit to show you really care about his or her business. Then recode the individual immediately so that he or she no longer is in the enterprisewide control group. You want to protect the statistical integrity of the universe, but not at the expense of losing a good customer.
You're not guarding state secrets. You're just measuring a marketing program, and recoding a handful of control group customers over the course of a year won't have any effect on the overall measurement.
This is a selection from “The Business of Database Marketing” by Richard N. Tooker. It will be published this month by Racom Communications (www.racombooks.com).
