I was stressing.
We had seen 40% of our customers cancel, and it was eating me up inside.
Each of those (former) customers had:
- decided they had the problem SocialWOD solves (workout tracking for every gym member at your gym, just by snapping and emailing a photo of your results whiteboard)
- decided they needed to solve that problem
- done research on solutions that might solve their problem
- decided that SocialWOD was compelling enough to try
- decided to take out their credit card to pay SocialWOD
- used and promoted SocialWOD to their members
After investing that much mental and emotional energy, they then they decided to cancel.
Argh.
We were trying to understand what the key differences were between our customers that cancelled, and our customers that didn’t.
We hop on the phone every week or two to call up customers, so qualitative data was no problem.
But we wanted to see what we could learn from our data.
Enter the Cohort Analysis
A cohort analysis helps you see how behavior varies across different cohorts of customers.
In this case, we were interested in the behavior of the cohort of customers who HAD cancelled, vs the behavior of those that hadn’t. What differences were there in each cohort’s usage of our product, and could we steer more gyms to do things like those gyms that had stayed on as customers?
I had read high-level stuff about cohort analyses, but it was this 90 minute cohort analysis class in NY with genius marketer Cassie Lancellotti-Young that really opened my eyes to how to run a proper one.
What Tools to use
Cassie recommended using Excel, and provided a template to use as the basis for a cohort analysis.
She mentioned tools like Kissmetrics and Mixpanel, but they were harder to use, less flexible, and required more overhead than Excel.
Given that likely all of the useful data you’ll want to analyze lives in your application’s database, her suggestion made a ton of sense to me.
How to start
We started by asking questions.
In this case, we were trying to answer this question:
“What are the major differences between customers that cancel, and customers that don’t?”
Pull some data
The next step was to write a ginormous SQL query pulling out all the data that could possibly help us answer this question.
Here’s some of the relevant data we pulled:
- The month a customer joined in
- The amount a customer paid per month
- Free trial length
- The number of a gym’s members that claimed their SocialWOD profiles
- # days after paying before canceling their service
- whether a gym uses SocialWOD to post their workouts to their gym’s Facebook page (where the gym’s members hang out)
All in all, there were 51 different fields of data that we pulled.
Analyze the raw data
I ran pivot tables comparing the difference between gyms that cancelled and gyms that didn’t.
I learned that customers who cancelled:
- had about half as many of their members claim a SocialWOD profile (so they could engage with our product)
- used the “post our workout results to my gym’s Facebook page” feature about half as much as gyms who remained customers
- paid about 23% more than gyms who stayed
- cancelled their subscription after 61 days on average
Square the quantitative with the qualitative
We have a spreadsheet that details cancellation reasons for every customer, and I’d estimate we’ve talked to at least half of the customers who quit.
The two most common reasons we kept hearing about why gyms quit were:
- our athletes aren’t using it
- it’s too expensive. SocialWOD is a nice-to-have (we’re working on that 😉 ). A member CRM is essentially a must-have for a gym over a certain size. The price for a member CRM is lower that it should be imho, and it sets a strong price anchor in the mind of a customer (i.e. “I *need* a CRM and it costs $X. I don’t *need* SocialWOD, but it costs $1.5X.”)
Given that the quantitative data suggested high member usage and lower price were big drivers of customer retention, the qualitative and quantitative data told a pretty strong story about what to do to cut down on cancellations.
Acting on the results
Here’s what we decided to do about each learning:
1. A cancelled customer had about half as many of their members claim a SocialWOD profile (so they could engage with our product).
Previously, our customers would tell their members to sign up for SocialWOD by posting an announcement about SocialWOD on their gym’s Facebook page, writing on their gym’s blog (read by most members), talking about SocialWOD at the gym, etc.
We figured there’s nothing easier to act on than an email describing benefits and a call-to-action. So we improved our onboarding to help a gym owner export a CSV of their members’ email addresses to send to us.
Once they do, we email each member a description of how SocialWOD benefits them, along with a link to click to sign up.
2. A cancelled customer used the “post our workout results to my gym’s Facebook page” feature about half as much as gyms who remained customers.
We improved our onboarding so that customers who haven’t set up posting to Facebook receive an email detailing the benefits, providing a video, and a strong CTA to do so.
3. A cancelled customer paid about 23% more than gyms who stayed
We built some technology than makes processing photos cheaper for larger gyms, and passed those savings on to both our existing and new customers. Those changes enabled us to drop prices by 15-60%, depending on price plan.
4. A customer cancelled their subscription after 61 days on average
We haven’t acted on this yet, but I can imagine sending gyms that meet some “likely to cancel” criteria an offer around day 45 of their membership to lock in for a year at a heavy discount.
The Outcome
Since implementing changes 1-3 two months ago, we’ve seen our cancellation rate drop from 40% to 5% – an 87.5% decrease. We’re going to run another cohort analysis in a couple months to isolate the impact of each change as it’s still too early to know the long-term impact of these changes, but in the short-term both Ryan and I are stoked about stopping the hemorrhaging.
—
Interested in learning how a cohort analysis can help your business grow? Get in touch – I work with select clients to help identify growth and retention opportunities, and build features to realize those opportunities.
—
Update (9/2/2012): You can now learn how to do a cohort analysis in this downloadable class.
Pingback: How we reduced our cancellation rate by 87.5% | Inbound.org
This is an awesome analysis. Kudos on improving the UX based on data.
Thanks Mehul. There’s gold in them thar data… you just gotta be able to shake it out from the noise.
Great play-by-play. A few questions though:
“Those changes enabled us drop prices 15-60%”
>> How has that effected your gross revenue?
And, did you go to any lengths to isolate the variables that you changed? In other words, perhaps the lower price-point doesn’t matter as much as those other components, so dropping your price by 15-60% may be unnecessary and is cannibalizing your revenue. Interested in your thoughts.
In this case revenue isn’t the most valuable metric, but profit is (or profit per customer or revenue:profit ratio). By reducing costs, reducing the price may necessarily reduce revenue but doesn’t necessarily eat into profit margins. Of course profits margins would be smaller than if they had maintained the same price and reduced their costs.
Still, I agree that this is an interesting statistic worth measuring and seeing.
agreed re: profit. it has gone up (customers signups have gone up).
as you might imagine, gross revenue declined but has recovered to pre-price drop levels (we saw an increase in signups). Also, we didn’t isolate the variables. Our hypothesis is that price was the biggest factor in seeing cancellation rate drop. If the answer could have a meaningful impact on our biz, I’ll run another cohort analysis several months from now to see if I can find some answers.
Kareem, great insightful post!
This is an excellent process you’ve followed to reduce churn which is killing businesses.
What I liked most, is that you’ve first identified the problem you have, gathered data in order to act on it.
Guy Nirpaz,
Founder of Totango
Customer Success Software
Charles Kettering said “a problem well stated is a problem half solved.” I try and keep that in mind before writing SQL 🙂
Hi Kareem,
I’m interested in your bootstrap cohort analysis pkg … When will you run the 50% off deal again?
Best,
JJ
Pingback: Flow » Blog Archive » Daily Digest for June 17th - The zeitgeist daily
Perhaps for #4, instead of just offering a straight-up discount, you could call customers around the 45-day mark and ask them how they like the product and whether there’s anything more they would like?
@rohan yep, that’s a possibility too. we usually call our customers around 20-30 days in (which is 35-45 including the free trial period) though.
mmmm … I would imagine everything comes from your price drop. The other stuff is good to read though.
that’s our hypothesis too. we’ll test it later if it makes sense.
I have to say, as impressed as I am with the scientific and methodical approach you took with analyzing the problem. I am a bit disappointed in your rather unscientific “shotgun” approach to fixing it. Perhaps you felt that time was of the essence and you have to stop the bleeding now, but it would have been interesting to solve one problem at a time and isolate the impacts.
Still… kudos to you for running your business on legitimate analysis.
thanks. while we’re curious about cause and effect of changes we make, our priority is building a profitable business right now. we can always try and isolate the effects if it will help us be more profitable later.
Kareem – Fascinating article. The 45 day offer and the common characteristics of cancelling customers is something I’ve seen at each software company I’ve worked at as well as something I’ve heard alot lately while talking to potential Klaviyo customers.
One thought on how you address it. First, perform the same cohort analysis you’ve already done, but look at the cancelling customers vs retained customers at day 1, day 15, day 30 and day 45, then use this analysis to figure out your triggers (things like # of Facebook posts needed by day 15, % of profiles claimed by day 30, etc). Once you have your triggers, you can make proactively calling / emailing problematic customers a key part of your daily routine. While discounts might still be the way to go, this trigger based approach is one we’ve seen work well for our clients.
Glad to discuss it in more detail.
That’s a great idea Ed, thanks for the suggestion. May very well reach out to you once we address that problem.
Inspirational success here, Kareem. I’d love to see a follow up where you give an intro on how to run a cohort analysis.
Thanks Gab. It’s on the list 🙂
Pingback: Audience and Product Experimentation « Tinker On
Hey Kareem,
Nice article. Your point about what to do on Day 45 rhymes with some of what I’ve been saying recently on the Intercom blog about retention.
http://blog.intercom.io/churn-retention-and-reengaging-customers/
Regards,
Des
Thanks Des. Your piece is great – definitely worth reading.
Great read Kareem. Any chance you can share the excel templates you used?
Thanks Tyler. I’m likely going to write another post about how to run a cohort analysis soon, and include the Excel templates with that. Stay tuned!
Pingback: Can you provide examples of following a data driven approach to product management in consumer web products? - Quora
Pingback: Monday Links from the Bench vol. CXXIX
Pingback: 192: TZ Discussion – Trillions and Trillions of Stars
Nice post. We’ve been using Kissmetrics but it’s tough to get a specific as you have. Have you thought of writing some analytics into the app so you can see how your cohorts are doing on the fly?
At most we’d find interesting changes week over week, but depending on what we’re looking for it’s month over month. Once the SQL query is written, it’s trivial to run and import into Excel… so I’d rather spend the time writing new features or growing SocialWOD than by building in-app analytics right now.
What Mehul said.
Thanks Weston!
Thanks for sharing, and congrats on that cancellation rate decrease!
Would love to know more about future changes.
Thanks Sylvain. Stay tuned 🙂
Nice post kareem.
would love to have that excel template with your next post so I can apply it on one of our project to get insights.
I may share in a future post Najam. Thanks for reading.
Pingback: Your Audience: To Lead or Listen? | Just Sherpas Blog
Pingback: Management Improvement Blog Carnival #170 » Curious Cat Management Improvement Blog
Do you have a twitter profile?
yep it’s http://www.twitter.com/kareem
Pingback: Viv Taylor » How we reduced our cancellation rate by 87.5%
Pingback: “This Is How We Built It” Case Studies (Part 2)
Pingback: 75 Instructive Case Studies - This Is How We Built It – Smashing Magazine
Pingback: 75 Instructive Design Case Studies | Design Guru