The Ultimate Guide to Control Groups

Want to make money yesterday? Embrace control groups. Here’s how.

So why’d Iran cease all air patrols on Friday?”

The airwing intelligence commander muttered the question without looking up from his clipboard.

I had just wrapped up my morning intelligence briefing. It was my first as a junior intelligence officer on board the USS Carl Vinson aircraft carrier.

We were steaming off the coast of Iran. Inquiring minds wanted to know why Iran hadn’t flown any military aircraft that Friday.

I moved towards the map on the wall. “Well sir, every day this week we have been patrolling here, right off their coast. And they have been flying intercepts. But on Friday we moved farther out to sea, so we are obviously less of a threat. They kept their planes on the ground.”

Wrong. And he made me aware of that in no uncertain terms…

“Listen, Lieutenant, there’s no ‘obviously’ about it. Friday’s holy day; they don’t fly. I suggest you hang around here and observe for a while before you toss out opinions, is that clear?”

Ass chewed. I had confused causality with correlation. Rookie mistake.

And you know what? As online marketers we do the same thing every day:

• Does your paid ads agency take credit for a 500% ROI on a branded ad campaign?
• Is your email person spouting off about a 1000% ROI on her win-backs?
• What about articles like this or this or this? Do they make you wish you could get the same “killer ROI” out of your own marketing?

These hot-takes all display the same rookie tendencies I made twenty years ago on board that ship. They trumpet the results of a marketing campaign without having investigated what would have happened anyway.

They screw up the causality.

How do you prevent this mistake? You rock control groups like a boss.

That’s the subject of this ultimate guide.

I know, I know, start the “NERDS!” chant … “control groups,” right? Who needs them?

Well, you do.

I am here to tell you that control groups can make you a lot of money, fast.

As in money that you are throwing out the window today. By showing you where to cut unnecessary spend, control groups make you profits overnight.

Read on, Nerd.

Your problem: your customers do stuff

Customers do stuff.

Sometimes that stuff includes buying things from you.

Maybe they buy because it is sunny outside or because their mothers laid on a guilt trip or because they want to get lucky this weekend.

Whatever the reason, you may have had nothing to do with it.

Think about that for a second. This creates a problem — how do you know if your marketing is working?

To recap:

1. You want profits.
2. To get them you need productive marketing.
3. To get productive marketing you need to calculate your ROI (ROMI).

But the last thing you want to do is to spend up on a campaign if you don’t have to.

Think of Adwords branded ad campaigns. Always super-high ROI. And your paid agency is oh-so-quick to take credit for that ROI, right?

But, ahem, do you really need to pay for those super-high ROI campaigns when potential customers are searching for your brand by name in the first place?

There’s no universal yes or no answer here. But control groups will help you to find the answer for your company—and save you money in the process.

How much? Dunno for you, but I can say that at Karmaloop we were spending six-figures on branded search each month. That’s a lot of moola.

What is a control group?

A control group is a group of people that we carve out of a promotion or campaign. Not with a knife. Meaning we don’t send or show it to them.

We send it to another group: the test group.

We then compare the behavior of the control group and test group. We want to know the lift in profits, if any, of the test group over the control group.

That lift is all that matters. That’s what determines your ROI.

Note that this is a special subset of A/B testing.

In a true A/B test, there’s an A variant and a B variant. Some customers see A and some see B.

In control group testing, the B variant—the control group—sees nothing at all. This is the only way to determine the lift that our campaigns produce over what customers would have done anyway.

The benefits of control groups

There are at least five killer benefits to using control groups.

Benefit #1: To find out when your ROI is deceptively high

News flash: Most online businesses doing six-figures and up spend heaps of money trying to get customers to buy when they would have bought anyway.

Identifying and eliminating that spend is the primary benefit of using control groups.

Subsidy costs: Is your entry popup losing you money?

Consider this scenario. My wife, Sara, decides to buy a new sweater at Club Monaco. She knows the exact sweater—the model number, size, price, everything.

She loads up clubmonaco.com on her iPhone …

… And before she can even add the item to her shopping cart, she’s hit with a 20% off popup.

Sara snatches the 20% off. Club Monaco Marketing is pumped because they see her convert. They calculate the ROI. Because that ROI is positive (even after the 20% discount), they call it a win and keep running the promo.

But that calculated ROI number is too high. It takes credit for converting Sara when she was about to pull the trigger at full price!

In reality, the ROI is lower, maybe even negative, because Club Monaco didn’t have to spend all those promotional dollars to get some of those conversions.

This hidden cost is called a subsidy cost, and I’ve written about it before. Reducing it is the quickest way to add profits to your bottom line.

Enter control group testing.

To test, Club Monaco would first carve out a control group. Again, this is not actually by using a sharp implement. The control group would not see the 20% off popup. If Club Monaco watched that control group, they’d see that there are women out there who come to the site, credit card in hand, and proceed to buy at full price.

Is the size of this “will pay full price” group large enough to offset the inevitable decrease in overall conversions? Testing would tell.

Armed with knowledge observed from this test, maybe Club Monaco ditches the offer. Maybe the company changes it up. Maybe it switches to an exit intent popup.

Don’t know, don’t care. All I know for certain is that with more accurate measurement, Club Monaco will pocket more dollars.

Benefit #2: To find out when your ROI is deceptively low

Promotions can change a customer’s behavior even after the promotion has ended.

Control groups help to measure that (delayed) change.

Win-back campaigns are a great example of this. Most of you send win-back promotions to customers who haven’t bought in a while, right?

You know the logic. These customers are defecting. If you wait too long to lure them back, it will be too late.

You gotta give these defecting customers a reason to buy again, so you send out some coupons and offers that expire within a week.

Because of this win-back campaign, some customers do indeed return. They take the bait. At the end of the week, you measure their activity. You slap an ROI on the campaign and call it a win.

But you know what? That promotion isn’t over! Far from it. Some of the customers you’ve won back will stick around for weeks if not months after your coupons expire.

These defecting customers are now back in the fold. Some will even buy again and again at full margin later on.

Glorious Halo Effects

This is called a halo effect. And it’s glorious.

This promotion and its glorious halo effect just juiced up your customer lifetime value.

But if you had stopped measuring your ROI at the end of that first week when the coupons expired, you’d have missed the halo effect. Indeed, your initial ROI on that win-back would have been low.

With a low ROI number, you might look at your ROI spreadsheet and dismiss the win-back in favor of a slick new Facebook Bot campaign.

You’d reallocate your time and effort and dollars away from the win-back when you should be doing just the opposite.

Control groups reveal the truth. By holding a control group out of your win-back campaign, you would see three things.

1. First, you’d see the control group defect as predicted (revenue per customer would go to zero).
2. Second, due to your win-back, you’d see the test group re-engage during your promotional week (revenue per customer would stay high).
3. Third and most surprisingly, you’d see the delta between the control and test groups continue for some time after the promotion ended.

You’d then include this long-term boost in your ROI calculation. Your final ROI would be much, much higher than your initial projection.

And with a more accurate number you’d then say, hey, win-backs kill it for us.

You’d want to lock them in, optimize the offer, and generate more profits.

Benefit #3: To calculate an ROI when there’s no response

How the hell do you measure things like your speaking on a panel at a conference?

What about that money party you held at Sweet and Vicious with the local vodka brand?

Where are those “brand-y” things in GA?

They aren’t there—but you should still try to measure them. They are costing you something, after all.

And if you want to keep throwing parties and appearing on stage you should justify these things.

Control groups can show you the impact of these non-direct-response events.

How Bonobos can justify a killer party with Jimmy Butler

Here’s how. When you send your local customers an invite to your next shindig, you leave a random sample off the invite list. You then watch to see how the invitees (test group) and non-invitees (control group) perform (i.e., buy, click, engage) over the next few weeks.

You calculate the lift. Voila, you can now slap an ROI on “brand stuff.” Now you know if the hipster parties are worth it.

And this ain’t just for ecommerce brands, BTW. SAAS readers: Constant Contact (5 million customers and growing) attributes their success to speaking at local chamber of commerce meetings.

Wonder how they figured that that was working?

Benefit #4: To negotiate attribution shit-shows

Case studies like this one make my head spin.

Online mattress company Casper is emptying the kitchen sink on social media.

After running this gauntlet of Instagram fake photos and content and in-store events and Facebook contests, some customers do indeed buy … but how the hell does the Casper CMO Michael Behrens know which specific tactics worked??

Is he just going to write the social media team a blank check and tell them to knock themselves out?

Is he going to pay Analytics Pros five figures to try to jam all that social craziness into a decent attribution model in Google Analytics?

(Don’t answer—he may actually do both, because Casper has VC cash. But you shouldn’t, because you don’t.)

How Casper can raise another $170m

No! He’s going to run control groups wherever he can!

By removing a random sample of his customers from this social media activity, he skirts the attribution issue. All he has to do is measure the lift of the customers who ran the gauntlet over those who did not.

There’s his ROI.

Armed with this ROI, he now can tell his board that Casper needs to go all Kardashian on social. And he’s then much more likely to raise another $170 million to pour into Instagram.

Attribution is tough. Simplify it with control groups.

Benefit #5: To make more money, full stop

All these benefits are about one thing: getting a more accurate ROI number on your marketing spend.

If it isn’t clear so far, I’ll say it again. The more accurate your measurements, the more money you will make.

This isn’t an intellectual exercise.

It’s not work for the sake of work or geeking out on spreadsheets.

No, this IS your job as a data-driven marketer.

You are the Warren Buffett of your company. The head allocator of your marketing dollars.

You make investments and back horses in the race.

The more you move your and your team’s attention, time, and money away from low-ROI activity to high-ROI activity, the more cash flow you will generate.

On a regular basis, you want to audit campaigns using a spreadsheet like the Nerd Marketing ROI Calculator. Identify what is working and invest in it.

Control groups (and testing more broadly) are your number one tool for beating the market.

Heck, this is why the former CEO of Harrah’s casinos, Gary Loveman, has stated that there are only three things that can get you immediately fired at Harrah’s:

1. stealing,
2. sexual harassment, and
3. running an experiment without a control group.

How to run a basic control group test

Sold on control groups yet?

If you’ve read this far, I’m assuming the answer is “yes.” So let’s dive into the nuts and bolts of setting up these tests.

As you probably suspect, you can’t just log into MailChimp and start pounding on the big “Control Group” button.

You can still, however, configure some basic tests fairly easily.

In the second half of this post, I’m going to walk you through how you might set up a very basic control group test using email software.

My goal is to make you money, today.

Spend an afternoon working through these five steps. You’ll end up with a campaign that generates profits forever.

Step #1: Pick a marketing campaign

You can run control group tests on almost any marketing campaign. But here are some things to consider:

Choose a campaign that is costing you margin

Consider two campaigns:

1. a Facebook ad campaign, and
2. a Facebook ad campaign that features a 20% off coupon.

I’d want to start testing with the latter.

The more promotions and discounts in your campaign, the better — for now.

Why? Because there’s a higher potential subsidy cost living in that campaign. Which means you may be giving customers an unnecessary discount.

Testing will show you if this is the case.

Choose a repeatable campaign

Consider two campaigns:

1. a one-off Valentine’s Day email (sent once a year), and
2. an ongoing email win-back campaign (sent any time any customer stops buying).

Opt for the ongoing campaign.

The more repeatable the campaign, the better.

Why? Because you’ll get more mileage out of your test.

Choose an email campaign

Finally, I’d start my testing with email campaigns. Email makes it easy (I’ll explain why), and I want you to get a quick win.

Bottom line? Start with your email win-back campaigns

For most of you, the perfect place to start testing is with your email win-back campaigns.

Win-back campaigns:

1. use discounts,
2. are repeatable, and
3. leverage email.

They are perfect for control group testing.

If you have a win-back campaign running, then use that for this example.

If you don’t have a win-back campaign running, create a simple one now. Here is a rule for a basic “one-size fits all” win-back:

Basic win-back: If your customer last purchased 60-120 days ago, send her a 20% off coupon to come back and buy.

Easy enough, right?

BTW, nerds refer to this customer segment as an “R60-120.” “R” stands for recency (days since last purchase) of 60-120 days.

As we are all nerds here, let’s roll with R60-120.

Action step: To start control group testing, consider your own win-back campaigns. If you don’t have any, use the “R60-120 / 20%” example I gave you above. Also, get ready to make some quick cash. Decide what you are going to blow it on.

Step #2: Create your test and control groups

Now that we have our campaign in mind, we need to create our two groups:

• The test group gets the offer,
• The control group gets nothing.

There are a couple rules for creating these test and control groups.

Rule #1: Your control group shouldn’t be too big. Or too small.

First, your control group should be about 10% of the total group of eligible customers.

The last thing you want to do is have your revenue go to zero because you dropped 90% of your customers out of a promotion.

That’s too much.

Dropping only 10% allows you to run a decent test without (potentially) shooting yourself in the foot.

In our win-back example, that would mean that I’d first pull a segment of all customers who are eligible for the win-back:

1. Say I have 10,000 R60-120 customers in my example.
2. I’d then want my control group to be about 10% of this.
3. That’s 1,000 customers (for those who can’t do 10% of 10,000).

That 10% can go up or down depending on how big your initial customer set is.

The smaller your total segment of customers, the larger that control group percentage needs to be. So if you only have only 100 customers, you need a control group of more than 10%—20% or 30%—to be confident in your results.

Conversely, if you have millions of customers in your segment, your control group can be smaller. Maybe 5%.

So start with 10% and work up or down as you see fit.

(And yes, I realize that statistics PhDs out there are ready to get all up in my nuts on this rule of thumb. But I’m a marketer first. Aim for production, not perfection. Let’s do this and not lose anyone in the process.)

Rule #2: Your control group should be random

Second, your control group must be random. Don’t skew your results!

What would skew the results? Well, choosing your control group on the basis of non-random stuff like:

• Total spend (“I’m just going to pull my list of customers, then sort by their spend and take the top/bottom 10%”)
• Number of purchases (“I’m just going to pull my list of customers, then sort by how many orders they’ve made with us and take the top/bottom 10%”)
• Recency (“I’m just going to pull my list of customers, then sort by their most recent purchase date and pick the top/bottom 10%”)

No sorting on these things! In fact no sorting!

If you were to generate a control group like this, that group is guaranteed to be better or worse performing than the test group. Your control group test will go right out the window.

No, we need your control group to be completely random.

Fortunately, there’s an easy way to do this in your email software …

How to generate a random sample in your email software

There’s no “random sample” button in Mailchimp or Klaviyo.

But most email software does offer a down-and-dirty work-around.

When I’m passing the time generating control group segments in my email software (“NERD!”), I like to create the segment off of the presence of a letter in the email address.

Most email software allows you to build segments off of text matching. I find that if I build a segment wherein the email address does not contain the letter “a,” for example, that that often runs about 5-10% of my total emails.

If “email address does not contain the letter ‘a’” is your control group, then “everything else” is your test group.

Good enough for most lists.

Here’s how you’d do this in Mailchimp:

And here’s how you’d do this in Klaviyo:

Got it? Good.

Action step: Create a win-back control and test group in your email software. Play around with the text matching to ensure that the sample generated is about 10% of your total list size. And like I said, the results should be random.

Step #3: Send the campaign, yo!

Now the fun part!

Send your campaign to your test group. Don’t send it to your control group.

(Fun, right? Hey, it’s a slow day here at Nerd Marketing HQ.)

I’m talking about a one-off send here. Although most email service providers are capable of automation, let’s hold off on that for now. We want to see what’s working before we lock it in.

For the R60-120 / 20% win-back example, our campaign would be a simple “Come back and save 20%” email. The email should contain a coupon for 20% off that expires in a few days.

In Mailchimp, you would want to send this email to a Saved Segment, which is your Test Group:

In Klaviyo, you would target your campaign like this:

Important: Once you send that single email campaign, do two things:

• Note the date. We want to look back at customer behavior since this time when we assess lift.
• Export snapshots of both your test group and control group. These can be simple lists of emails. You want to log which customers were in each group for measurements later on.

Action step: Send your own win-back offer to your test group. Note the date and export the lists of test and control group customers.

Step #4: Measure the results

Now the not-as-fun part!

You want to measure the difference in profitability (if any) of the test group over the control group. This will tell you whether the campaign was worth running.

Some tips on how to measure this profitability:

Tip #1: Measure profits per customer

You want to measure profits here, not revenue.

Why? To keep ramming this point home, it’s because you used discounts in your campaign. You therefore need to see whether that discount margin was worth giving away as a promotion.

Moreover, you want to measure profits per customer. Obviously, your test group is much bigger than your control group. So the test group will have more total profits.

To compare apples to apples, you need the per-customer metrics.

Tip #2: Measure only after90 days

Remember: The expiration date on your coupons is not when the promotion ended.

You probably have a glorious halo effect. You should wait longer—90 days is a good rule of thumb—to measure how your campaign performed.

Why 90 days? Because when you send a coupon, you alter customer behavior for a while. Think about it:

• Mary is defecting, never to order again.
• You send her a coupon, she takes the bait and buys on discount.
• She loves her purchase so much that two weeks later she comes back and buys at full margin.

Pre-win-back, Mary was out of here. Gone. Post-win-back she’s ordered not once but twice. And the second time was two weeks after the promotion ended. And it was at full margin.

This is your glorious halo effect. You won’t have 100% Marys, but you’ll have some. You have to attribute them back to the original win-back, even if the coupons expired weeks ago.

So wait 90 days before counting up your profits per customer. And when you do, don’t count coupon redemptions. Count total profits per customer on all orders from the control and test groups during that 90 days.

Tip #3: Use a spreadsheet

It would be nice if your email software did all this measurement on the fly, right?

Profits per customer per segment group over 90 days?

Unfortunately, I have only seen this kind of sophistication in super-primo email software.

That said, it’s easy enough to run these numbers in a spreadsheet. And much cheaper too. I’ll walk you through it here using what I call the Convoluted Spreadsheet Method.

(OK, you actually can start the “Nerds!” chant now.)

Drew’s Convoluted Spreadsheet Method for measuring test performance

You can read through here and if you get stuck watch my video tutorial:

1. Export your transactions from your shopping cart. Import them into Google Sheets. Call the current tab ‘Orders Export.’ Here’s how this looks in Shopify:

   

2. Only look at orders that occurred since you sent the win-back (i.e. the 90 days since the date you marked in Step #3 above). In Google Sheets, sort the ‘Orders Export’ tab by ‘Created At’. Delete all orders from before the date you sent your win-back. Now you are just looking at the post-campaign results.
3. Only look at customers in your test group or control group. You have those “snapshot” lists from Step #3, right? Import both lists as a new tab in your sheet. That sheet should have two columns: one for email, and one for Group (‘Test’ or ‘Control’). Sort by email address. Now use the VLOOKUP function to identify which customer emails from your ‘Orders Export’ tab are in either your control or test group. (I recommend creating a new column in your ‘Orders Export’ tab for ‘Group.’ Set the value there to either Test or Control or (blank).)
4. Re-read the step above. I mean, really start the NERDS chant now if you haven’t already.
5. Delete out the rows where the ‘Group’ is blank. Now you are just looking at customers in your control and test groups over the 90 days since you ran the test. Your sheet should look like this:

   

6. Add a margin dollars column in the Orders Export tab (‘Margin ($)’). Keep it simple, something like Margin ($) = Revenue – COGS – Discounts. Calculate that for all orders remaining in your Orders Export.
7. We want to know the profits (margin dollars) per Group. Pivot table anyone? Create one that sums up the Margin ($) for the test and control groups.
8. To get to profits per customer per Group, take that margin number for each group from the last step. Divide it by the total number of unique customers in each group. Your pivot table should look like this:

   

9. Calculate the lift of the test group over the control. What’s the difference between the two per customer per group totals?

Piece of cake, right?

If you live in Google Sheets, it actually should be straightforward.

But if you’ve never done LOOKUPs or pivot tables before, it might take you some time. To help you, watch the video below. That might help.

Or, better yet, hire someone cheap on Upwork to do this for you. All this cranking in Google Sheets looks time-consuming, but it should take a spreadsheet nerd 10-15 minutes.

Value your time and outsource.

Tip #4: Calculate the ROI

Profits per customer per group tells you which won: the control or test group.

Want to get fancy? To calculate the true ROI (if any) of the test campaign, take the lift of the test over the control. Plug that into the second tab of my Marketing ROI spreadsheet.

Remember: Don’t get greedy. Your ROI is only the lift of the test versus the control. It’s not the absolute ROI from within the test group.

Action step: Measure the results of your campaign using the Convoluted Spreadsheet Method. Or hire someone to do it for you.

Step #5: Retest (or don’t)

At this point, you know whether your win-back campaign makes you money.

You know whether you have been spending too much too soon to keep customers around.

For most of you, I’d guess that the answer is yes, win-backs are worth it. The profits per customer of the win-back test group typically beat that of the control group.

If this is the case for your business, go ahead and lock in that campaign. Set it up to run automatically using your email software. Extend it to Facebook using custom audiences. Mail it out via PostPilot if you feel up to it.

You can be certain that the campaign will generate solid profits per customer for you for every customer you run through your funnel going forward.

If, however, your control group beat your test group, or if you want to keep testing different offer levels or timing, take a step back and rethink things.

• Was the promotion too aggressive or not aggressive enough? Maybe you had a high response rate on your test but profits per customer went down. Run the test again with a lower discount.
• Was the timing off? Maybe 60 days was too soon: try 90. If you go too aggressive (early) in with your win-backs, you are sure to start increasing that subsidy cost.

Action step: If you have a winner, lock it in. If not, run the test again with different promotions or timing.

Where you go from here with testing

Sold on control group testing?

Here’s where you might go next…

Up your game

If you have enough customers, you can get more sophisticated with your tests.

At Design Public, I was always combining control groups with my A/B and A/B/C tests.

So in the case of a win-back, I ran something like this:

• Control group – no offer for R60-120
• A test group – 10% off offer for R60-120
• B test group – 20% off offer for R60-120

Once I found the ideal offer, I’d play around with timing while holding the offer constant:

• Control group – no offer
• A test group – 20% off offer for R30-60
• B test group – 20% off offer for R60-90

With enough testing, I could tell which offers and timing were best for my business.

This would allow me to build a hyper-dialed-in discount ladder that maximized profits.

Isn’t that cool? I mean, it doesn’t get any better than this with database marketing. After years of testing you could literally have a customer retention machine just cranking for you while you sleep, throwing off maximum profits left and right.

If you are starting out, however, keep it simple. Hold variables constant. Limit yourself to one control group and one test group per test until you get the hang of it.

Push to other channels

One reason I recommend starting with email is that your email software can serve as your one-source CRM.

Create groups and segments there first. Then push them to other channels.

• Push them to Facebook as custom audiences using an app like Plus This.
• Target them on your front-end using some javascript or an app like Right Message.
• Upload them to a snail mail service like Modern Postcard and mail them.

Out of the box, these other platforms and channels aren’t always set up for proper control group testing. But if you start with email, you can extend that testing to other platforms making your life easier.

Test on other key campaigns

Win-backs are a great place to start, but think about where else you might be bleeding cash.

For most online brands, I’d start with these:

• Branded ad campaigns in Adwords
• Entry pop-up offers
• Offers you have in your email Welcome Series

Not all of these are as easy to set up, but they can be done. Weigh in on the comments on other examples you might have.

Wrap up

I started this email on an aircraft carrier. I’ll end it in the Department of Defense.

Former Defense Secretary Robert McNamara once said, “Measure what is important, don’t make important what you can measure.”

Think about that. Don’t blindly pursue certain campaigns just because the numbers are readily available to you. (I’m looking at you, branded paid ads.)

What’s really important? The things that will generate profits.

And you can’t figure out what those things are until you make control groups your best friend.

• They allow you to reduce subsidy costs
• They allow you to measure halo effects
• They allow you to assess random-ass hipster parties
• They allow you to skirt attribution hornet’s nests

Use ‘em today. Watch my video tutorial. Work through the action steps in this post.

The more you use these, the more you know about your business. And the more you start to build a highly-customized marketing machine that generates profits.

Now—over to you. If you are using control groups in your current campaigns, can you comment below and let me know how? What’s working and what isn’t?

Got a comment? Email me at drew [at] nerdmarketing.com!