Jack Murray - RamblingsFocus your testing efforts with the growth stage matrix1635 words (approx 9 min)
Here’s the basic framework I use to think about growth stage maturity, and in particular the kind of 'growth' or product work that I should be focusing on at any given time.
It applies at both the product-as-a-whole level, as well as an individual-product-or-feature level. For example, it applies for looking at Spotify as a whole, and the Music or Podcasts or Audiobooks inside Spotify level.
Feel free to use this matrix for helping you to decide what kind of growth efforts your company or product team might want to focus on.
Don't worry, I'll talk through how to apply it.
Product Market Fit (PMF)
You can map where your product or a given feature in your product sits on product-market-fit.
Measuring product market fit is difficult, and it’s not a binary yes/no thing, which is why it’s reflected on a scale. The higher the better of course. You want to be in the blue highlighted segment.
You likely have a higher product market fit if:
- Your customers are very happy to pay for your product/feature
- Your customers are highly engaged (retained) on your product or feature
- You have strong word of mouth and a lot of your customers come through unprompted referrals from others
- Your customers genuinely rave about your product online to strangers
- Your Net Promoter Score (NPS) is above +50
- You can run the Sean Ellis Test (made popular in his book ‘Hacking Growth’), which is a single question survey designed to understand how many of your customers consider your product a must-have. The question is “How would you feel if you could no longer use [product or feature name]?” - with the answers being “Very disappointed”, “Somewhat disappointed”, “Not disappointed”, “I no longer use this product”. Where 40%+ of respondents answer very disappointed, you’ve got yourself a good indication of PMF. This should be split between the following qualifications (experienced core product or service, done this at least twice, and done this in the last 2 weeks).
In plotting where your product overall or your product feature is on this spectrum, take an honest look at the data, not just you or leaders’ opinions. You need to be very customer-centric on this, and you can’t just call PMF based on a single one of these metrics.
Be honest with yourself.
Luckily it’s not something that can be ‘scientifically’ measured, so a best guess is the best you’re going to get anyway.
Collate a different views, ensuring that they aren’t biased by others.
Customer Volume
Customer volume applies in slightly different ways relative to the purpose we’re thinking about it for. That’s because we can use this matrix for two main purposes. We can use it for determining the type of testing you probably want to do, as well as determining a focus area you probably want to think about improving (acquisition, engagement or monetisation).
For starters though, let’s just think about the customer volume spectrum as the actual number of customers using your product.
As a good rule of thumb, daily users is a good metric to use here for most software products. This is because there are certain types of testing where you need a sufficient volume for validly running tests, such as A/B testing. Daily users is good because you can more accurately guess how long such tests will need to run in days.
The bottom end of this spectrum is obviously no users, and the top end is in the millions of daily users. Generally, to be in the blue highlighted section for this you want to have something like at least 10,000+ daily active users.
Before you say it, because I know you’re thinking it… Yes, you can for example do A/B testing with just single-digit thousands of users, but you’ll need to run fewer tests or tests for much longer. This can become a bottleneck. Let’s avoid that for now.
Bringing it together and determining action
Once you’ve plotted yourself on both measures, you get a helpful decision making framework for thinking about the type of tests you might want to be the bulk of your testing, and the kind of directional focus you might want your testing efforts to focus around.
Overlay your two measures and you get the most relevant kind of testing for your product or business.
Lets walk through each of the segments - what they mean, what kind of work is most valuable to focus on, and some examples of each.
Find Signal
You’re likely in the early stages of your business, product, or some feature you hope to be core one day. You don’t have great PMF yet and you also don’t have a high volume of customers.
Kinds of testing to run
Because you don’t have the volume to do statistically significant tests, you need to rely on what many call “discovery” or “innovation” testing. This comes in the form of things like:
- Commitment testing (sales before building anything)
- Launching a minimum viable product (MVP)
- Riskiest assumption testing (RAT)
Metrics to focus on
Engagement is likely priority one, in the form of activation and retention. This gives you the signal you need that you’re on the right track. Depending on the business/product though, Monetisation or Acquisition could also be important.
Examples
- Early-stage start ups
- New ventures / products (e.g. a second app, or a new major product on an eCommerce store)
- New features that are currently ancillary to the core value proposition (e.g. Spotify launching Audiobooks alongside their Music core value prop)
Danger
If you’re in this zone and you’re not a network effects product, then you’re in an icky spot. You’ve got a lot of users but you still have low product market fit for now.
This is not a place you want to be optimising your product in, this is a place you want to be taking bigger swings in to up your PMF as soon as possible.
Many companies that find themselves here are VC funded startups that have been prematurely marketed. The other kind of things that end up here are new features or products launched alongside a core product offering.
Kinds of testing to run
The silver lining to your situation is that you’ve got enough users to do statistically significant tests. However, this is not time for optimisation - you need to be taking bigger swings to get your PMF up as soon as possible.
- Bold A/B testing
- Core value proposition testing
- Riskiest assumption testing (RAT)
Metrics to focus on
Engagement is likely the priority here, in the form of activation and retention. Try to monetise something without PMF and you’ll be pretty disappointed. Acquiring more users is likely wasted money as your churn is probably very high.
Examples
- Start ups with big marketing budgets
- Network effects products that haven’t reached their critical mass yet to achieve PMF (e.g. a marketplace product that doesn’t have enough demand in order to drive better supply yet)
- New ventures / products launched that drive traffic from another core product (e.g. Facebook’s “Threads” app in 2024)
Qualitative Testing or Acquisition
Here, you have low customer volumes but relatively high PMF. The low customer volume could be an active choice (luxury brands), a fact of the market you’re in (there’s just a low number of customers out there, e.g. certain B2B markets), or you’ve found PMF but have yet to really scale it.
Kinds of testing to run
Here you don’t have “on-product” volume to do statistically significant tests - that means you need to lean on qualitative methods.
However, if your goal is to drive Acquisition from here, you can “borrow” scale through paid ads. That means doing advertising experimentation on paid platforms like Facebook, Instagram, Tiktok, Linkedin, and so on. That allows you to run statistically significant tests despite not having your own sufficient volume.
Qualitative testing:
- Customer playback and workflow analysis
- UX hotspot and journey flow testing
Borrowed Scale testing:
- Marketing funnel testing
- Customer acquisition cost optimisation testing
Metrics to focus on
Acquisition is likely the priority here. Monetisation could also be your focus though, especially in the case of being a low customer volume business (like B2B or luxury). Engagement may be focused on but it’s less likely to be your primary concern.
Examples
- Start ups that have found good product market fit
- Luxury brands (e.g. Ferrari)
- Many B2B products (e.g. enterprise software)
Optimise or Expand
You have great PMF and high volumes of customers here. This is most successful consumer products. It’s where you are able to do what most consider to be ‘growth’ work by default. There’s a few choices you have here really:
- Optimise or expand the current value (improve acquisition, engagement, and/or monetisation)
- Expand via adding new value (like launching a new feature or secondary product) - though a new feature or product launched would likely then be split out and end up in the Find Signal testing.
Kinds of testing to run
You’re able to do all kinds of testing in this zone, and A/B testing volumes are easily satisfied.
- A/B testing
- A/B/n testing
- Multivariate testing
- Riskiest assumption testing (RAT)
- Pricing experimentation
Metrics to focus on
Any could be the primary measure to focus on - it really depends on your strategic direction. Sorry, I know that’s not a great help considering most people reading this will probably be in this bucket.
Examples
- Consumer scale-up companies (e.g. Skyscanner, ClearScore)
- B2B scale-ups that have high customer volumes and usually self-serve (e.g. Stripe, Amplitude)
- Mature products and companies with millions of users (e.g. Booking.com, Facebook)
- New product features that are now core and have high customer volumes using it (e.g. Instagram Reels)