subject: The What, When and How of direct mail testing [print this page] The What, When and How of direct mail testing
What to test
In his book, Secrets of Successful Direct Mail, Dick Benson states, "Any idea you honestly believe can economically increase response is worth testing."
The key words are "economically increase response." But what is economical? Typically, the more dramatic a change you make in a package, the more dramatic the difference in results.
For example, when you need a breakthrough, test the components that have the greatest influence on the mailing's successlists, offer, format and copy. Forget about testing minor changes on page three or the color of the return envelope. Test big things for big results.
However, for clients mailing larger volumes, "tweaking" the control for incremental gains often makes sense.
For example, Client A and Client B both have an average response rate of 1%, with a $25 average transaction. Both test a new package that lifts results by 10%. The only difference is that Client A has an annual mailing volume of 1 million pieces. Client B mails 10 million pieces every year.
For Client A, with its smaller mailing volume, this 10% lift means an annual difference of $25,000 in income. But for client B, the same 10% lift means $250,000 in additional income.
While the smaller client might appreciate an extra $25,000 to spend, it is probably not going to have a major impact on the organizationespecially after accounting for test costs. On the other hand, a quarter of a million dollars is "adult money."
But whatever you decide to test, do the math first. Even if the test beats your control, it won't make sense if it takes too long to recover your test costs or requires an unobtainable rate of response.
When to test
The best time to test is when you have a successful control. A too-common mistake is to delay testing new concepts because "our control is still working." By this reasoning, you wouldn't test new concepts until the control takes a dive, and this is a disaster for cash flow. You may even find that it's time to update your rsum, because there's not going to be enough cash flow to make payroll.
Instead, test from strength, not weakness. Virtually every mailing is an opportunity to test. However successful your control is, and however long it's been the control, it can be improved.
How much to test
When deciding how many pieces to test, the important thing isn't how many pieces you mail with your test package. What's important is how many responses you get back.
Your test volume should be no fewer than the number you need in order to produce statistically reliable results. Mathematicians using formulas I won't even pretend to understand tell me that as few as 39 responses can be statistically valid with a 90%95% probability of accuracy. Personally, I'm more comfortable with closer to 100 responses.
You can use an online calculator to tell you how much to mail, but the concept is easy. Just take the control's response rate and see how much you need to mail in order to produce 100 responses. For example, with a 1% response rate, you would need to mail 10,000 pieces to produce 100 responses.
As a rule of thumb, you'll want to limit your total test volume to no more than 20%25% of the total mailing. This way, if the test fails, the returns from your control will help absorb the loss and preserve the operating budget.
Smart testing
Testing is essential to profitable direct mail, but we need to test smart. Evaluate the success of the test based on its rollout cost, but always do the math first and make sure you can pay back your full test costs in a reasonable time.
And remember Dick Benson's advice and test only what you honestly believe can economically increase response.
