Everybody Jumps on the Metrics Bandwagon

JupiterResearch analyst David Daniels is spearheading an industry-wide effort to standardize e-mail metrics, starting with defining terms around delivery.

So far, just about everybody who’s anybody vendor- and trade-group-wise seems to have signed on.

Dubbed E-mail Measurement Accuracy Coalition, the initiative comes on the heels of various reports revealing the dismal state of e-mail metrics. According to recent surveys, many marketers are measuring little that is meaningful in their e-mail programs, and those who are, define fundamental terms vastly differently.

For example, the delivery rate—probably the most fundamental metric in the business—has no standard definition, according to a survey conducted earlier this year by the Email Experience Council, a for-profit organization aimed at promoting standardization in the commercial e-mail industry.

Seventy nine percent of e-mail service providers surveyed defined “delivered” by deducting all failures from total mailed, while 21% calculated it by deducting hard bounces—where the address no longer exists—from total mailed.

E-mail marketers were in even further discord on the “delivered” metric, as 63% defined delivered as total failures subtracted from total mailed, 11% defined it as simply total mailed, and 10% defined delivered as only those e-mails that made it into the recipients’ inboxes versus their spam folders.

As a result, standardizing deliverability metrics is the E-mail Measurement and Accuracy Coalition’s first task, according to Daniels. He added that rectifying some of the channel’s inherent sloppiness will be critical.

“A lot of the information that comes back from the ISPs is not accurate, and they do that purposely to throw off spammers,” he said. “So part of it is getting a … measure for how you deal with that noise, and then more importantly, what do you do with all the hard bounces and the failures? Do you keep them in the arithmetic or do you remove them?”

Currently, it’s difficult for marketers to judge the performance of various e-mail vendors because the service providers present their metrics differently.

“Clients have come to us and said, ‘We have some of our mail with this e-mail vendor and some of our mail with this [other] e-mail vendor and we can’t even internally do benchmarks because the two vendors use different measurement methodology,’” said Daniels.

He said he understands the idea that some e-mail service providers will have to change the way they measure their services may mean the process won’t necessarily go smoothly.

“It’s going to be a challenge to build consensus,” he said. “There are a lot of competing interests and there are some real costs involved.”

However, Jordan Ayan, chief executive of e-mail service provider SubscriberMail, said any changes involved in reaching an industry consensus should be painful only to vendors who have been fudging their numbers.

“If you’ve been very clean in the way you’ve been reporting data, doing whatever comes out of this will be relatively easy,” he said. “But if you’ve been doing tricks behind the scenes, you’re going to have to come out from behind the curtain. This is long overdue.”

Ayan added: “There are a lot of people who have very good delivery rates in this industry now. But there are also people who have bad delivery rates who have been reporting good delivery rates.”

Participation in the coalition is open to anyone and free, said Daniels. He added he expects the group to finish its work and disband by the end of summer. “This is not going to be yet another ongoing coalition,” he said.