When You Should Skip the Data Driven Marketing Control Group
There are times when you’ll be forced to abandon the idea of a measurement strategy using control groups for your database marketing campaigns. Despite your best intentions, there are situations in which it just doesn’t make sense to hold out a control group. Sometimes this is caused by a statistical issue. Other times it’s dictated by larger corporate priorities.
Small target audiences
Some marketing campaigns are just too small to allow for a useful control group design. Small means that the target audience isn’t big enough to give you the kind of precision you would need to do a meaningful analysis of your campaign. You need to be aware of one situation in particular.
If your geek comes back to you recommending a control group size that is getting close to half your overall audience size, then you might be in trouble. Whenever you’re setting up an experiment, the recommended control group size applies to not just to the control group.
It applies to the group you’re measuring as well. If your control group needs to include 10,000 customers, then that implicitly means that you need to mail at least 10,000 customers as well.
You need to be aware of this fact because your geek may perform your control group sizing before your mail counts are final. You perform a number of steps before you send out a mailing which have the effect of purging customers from your target audience. There are cases where this purging of records has dropped the actual mail file below the size of the control group.
Control groups come with a cost. To the extent that a database marketing campaign is successful, anyone you don’t mail represents a lost opportunity. In other words, control groups mean lost revenue.
The pharmaceutical industry provides an insightful example.
Some drug trials are wildly successful in demonstrating the effectiveness of a treatment. In some cases — especially ones involving particularly nasty or even fatal conditions — this success creates a moral conflict between the science of medicine and the practice of medicine.
In these cases, it is generally considered wrong to continue the drug trial and withhold an effective treatment from the control group. The trial is then suspended, and patients who had been receiving the placebo are given the actual drug.
Most marketing executives buy into the idea that the information that comes from well-designed tests is worth the cost in lost revenue. But cases will arise when this is not true.
One situation where it really is a little pointless to keep holding out control groups is for well-established marketing campaigns. If you’ve been running the same campaign to the same audience year after year, you may already have a pretty good idea of how much that campaign is driving to the bottom line. There really isn’t a compelling reason to keep verifying that contribution.
Control groups often go out the window in times of trouble or heightened concern about company performance. If your CEO gets concerned about the company meeting its quarterly or annual sales goal, your marketing executives will feel the pressure. And they’ll do everything in their power to squeeze as many sales as they can from their marketing budgets. Control groups are an easy target (pardon the pun).