Why E-mail Benchmark Studies are (Mostly) for Idiots

Any e-mail service provider knows that if there’s one sure way to get a marketer’s attention, it’s by publishing benchmarks.

Trade publications—this one included—jump on anything purporting to measure the nation’s e-mail open, click-through and conversion rates. After all, marketers want to get an idea where their e-mail campaigns’ performance stands in comparison to the rest of the industry, and as a result, have an unquenchable appetite for articles on the subject.

However, the studies are all but useless for determining much beyond the most general trends, according to John Rizzi, president and CEO of e-mail service provider e-Dialog.

For one thing, every business is different. As a result, nothing in a benchmark study will apply to any individual business, said Rizzi.

“No one’s business is a benchmark business,” said Rizzi. “What each one does is unique. They do things at a granular level that can be very different from aggregating 50,000 e-mail campaigns from 100 different people.”

Moreover, so many factors affect e-mail metrics that it is impossible to get useful benchmarks by aggregating a bunch of different campaigns.

“Is [the address from] a brand new member to your list? Or is it an old member? Is it an acquisition e-mailing, a newsletter, or is it a welcome message?” said Rizzi. “Welcome messages get tremendously good open rates. If you throw that into an aggregate, you’re going to raise the benchmark. But is it relevant to what you’re doing on any given day? Not at all.”

Also, he said, not knowing certain details about how the benchmarks are calculated muddles the numbers even further. For example, has the average been weighted, where smaller campaigns are given less “weight” than others based on their size?

“In an unweighted average—which is probably how most people do it—that welcome campaign that got really high open rates is going skew your benchmark high even though welcome campaigns are small,” Rizzi said.

Even within a sector, benchmarks are all but useless, he said. For example, in publishing, one newsletter may be self-contained and not require readers to click out of it while another may encourage people to click.

“As a result, which one is a better performing e-mail is an irrelevant question,” said Rizzi. “Benchmarks are not about comparing apples to oranges. They’re about comparing apples to oranges to blueberries to pears.”

To top it off, individual campaign metrics are easily manipulated, making benchmarks’ usefulness even more questionable.

“If I take only the customers who have bought from you five times in the last 90 days and spent $100 each time, and respond to great offers, I can get you one kickin’ response rate,” said Rizzi. “You can blow a benchmark anytime you want by taking your multi-buyers and sending them a great offer. Benchmarks don’t take any of that into account.”

The only metrics marketers should consider using are those from their own efforts, said Rizzi. “The important thing about e-mail metrics are ‘what are your goals,’ and ‘is this week’s campaign better than last week’s campaign?’”

So what are benchmarks good for? They can be helpful for figuring out if trends in an e-mail company’s efforts—such as a yearlong drop in open rates—are part of a national trend or something unique to that company.

“They’re not useless if the marketer’s smart enough to know what’s going on,” said Rizzi. “They might be pointers for the industry at large.”