Test Cell Traps
How to determine exactly what you’re testing ONE OF THE defining characteristics of database marketing is the ability to test and evaluate. By refining strategies based on “real-life” experiments, companies can apply more potent tactics in their quest to strengthen customer relationships.
A typical test program involves selecting segments, choosing samples, communicating and evaluating. Communication can occur at a variety of touch points. Segment identification is frequently determined through an analysis of previous programs, results from a model, or as many prefer, a `gut’ or hypothesis based on a subjective evaluation.
The segments that are used as test groups are usually “cell coded.” These codes are then used to identify the segments, and subsequent performance of each of the groups can be measured. While choosing cells for testing appears to be a straightforward matter, many analysts err in the actual selection and/or analysis of these cells.
For example, take a marketer of communication devices that is offering a new cellular phone. (Although the story line presented here is real, all figures are hypothetical.) Three test segments are isolated. The three groups to be evaluated are:
1. People age 25 to 34, who are college educated and married.
2. Customers in households with average monthly bills exceeding $100.
3. Small business owners.
The sample sizes used can be seen in Chart 1 on this page.
The sample process this firm uses is based on nth-name selection criteria. Essentially, a number or “n” is chosen, and every nth name that meets the criteria is selected for the program. There are a number of approaches in choosing this “n” value. Many of them work adequately. Some analysts demand that in order to further maintain randomness during the sample selection process, the counting of the “n” does not commence at the beginning of the file, but rather begins at some random point in the customer database.
Let’s clarify this a bit further. Suppose there are 5,000 customers, each one numbered 1 through 5,000, and they reside on the file in consecutive number order. Further, assume a sample of 1%, or 50, is needed for a test. One approach is to begin selection at number “1” and select every 100th account. At the end of this procedure, numbers 100, 200, 300, 400, 500 and so on will emerge in the group of 50 needed.
Another routine, preferred by some, is to start at some random point and then choose every “nth.” This random point may be secured from a table of random numbers. Assume, for example, this random point is 72. After this point, every 30th customer is selected (the analyst selected 30 as his “n” value – it may very well have been another number). Using this formula, the final resulting sample will include the numbers 102, 132, 162, 192, 222, etc., and so on until the needed 50 (1% of 5,000) are attained.
Our telecommunications executive now directs his attention to the first test cell, “1.” He uses one of the aforementioned methods to pick his 50,000 sample. After this is completed, the segment is identified as “Cell Code 1.” He then proceeds to populate his second test cell with 50,000 households. After assigning “Cell Code 2” to this group, he proceeds to choose his last segment of 54,500, and allocates these business owners to “Cell Code 3.”
After completion of the program, the analyst collates the results. These figures appear in Chart 2 above.
A conclusion could very well be drawn from this analysis that Cell Code 2 outperformed, in a rather marked way, the other two segments. And this statement is not incorrect. However, marketers must establish what their goals are in designing the test to be certain.
Although not intuitively apparent, there are seven possible segments that could be derived from the three cells noted here. A single account could conceivably have data characteristics that are unique to one cell, or are shared by one of the other cells, or appear in all three cells. These are summarized in Chart 3 on page 66.
The seven rows in this chart refer to the various combinations of characteristics that are available. The first row describes households that fall into Cell Code 1 – they do not have any of the characteristics that describe those in the other two segments. In other words, in addition to being “between the ages of 25 and 34, college educated and married,” no one in this row had “bills exceeding $100” or was a “small business owner.” The fourth row, on the other hand, refers to customers that have attributes common to both Cell Code 1 and Cell Code 2. That is, they are”between 25 and 34 years of age, married, college educated” and “have bills exceeding $100.”
What does the marketing manager want to test?
1. Is the objective to ascertain which of the cells do best, regardless of any overlap?
2. Or is the ultimate goal to examine households with the specific characteristic, excluding the other characteristics being tested?
Lest you believe there really isn’t any difference, you need only look at the response rates associated with each combination of attributes. See Chart 4 at right.
The response rates are clearly different from what our original analysis demonstrated!
Besides reaching wrong conclusions, if overlap is a consideration, experimental design becomes more complicated and requires additional planning.
For example, if many test cells are anticipated, there’s a possibility that some of these segments may not be random at all. The design procedure frequently begins with selecting Cell Code 1, then proceeding to Cell Code 2, then onto succeeding segments. However, the earlier selections may very well have siphoned off households that would have met criteria for subsequent cells.
Thus, are these later-selected cells really random or are they skewed by a diminished pool?
While database marketing offers the luxuries of testing and refining, the ultimate decisions must be made on a sound analytic footing. The true benefits can only be achieved with a careful research approach.
Sam Koslowsky is vice president of strategic analytics at Harte-Hanks Inc., New York.
