Viability

Once you’ve validated that a customer segment exists that feels deeply about the pain or passion you’re addressing, you might start running experiments to test (1) customer believability; (2) product fit; (3) business-model viability.

The data you collect depends on the test you run. Using ideas from the previous chapter or those that you dream up, run multiple experiments that expose problems with your segmentation, product ideas, or business-model assumptions. Continue to interact personally with customers in order to calibrate the quantitative data you’re collecting from the experiments.

Each time your pattern breaks, you’ve likely identified a different subsegment. You want to track your data by subsegment, by bucket. This is how you learn to scale. Assuming that your customers are fungible and homogenous, results in problems down the road.

Recall our segment matrix:

As you build product or create a viability test—for example, a Wizard of Oz test in which you pretend the product is done, but you actually fulfill it manually—you might find that some customers you thought were in Segment 1, actually don’t feel the pain deeply enough to sign up for your product. Should you put them in a different bucket? Do they fit in Segment 2?

Customer interaction and experiments are conducted to uncover and mitigate business-model risks, including most fundamentally, is the problem you are addressing worth solving from both your and your customer’s point of view, and is it solvable?

Surveys can help corroborate segment assumptions on a larger scale, but should not be relied on solely.

When the data indicates that yes, the problem is worth solving, you’ve reached the point of building product. Now’s not the time, however, to go back inside the building until the product is done. All through the product development cycle, you should continue to interact with customers.

Product-Market Fit

Imagine you have a group of customers you have been interacting with on a regular basis, and you have a deep understanding of them. You continue to interact with them on two levels:

1. You validate how you can acquire more of the same.

2. You validate that the product you are building addresses their needs.

For the first, you begin to run tests on acquisition channels and sales funnels² to ensure that you understand where this particular market segment hangs out, who influences their decision making, and what you need to provide so that they move through the emotional states of the buying process.

To measure, again start with a hypothesis:

“Posting my beta invitation to Hacker News will result in 1,000 visits and 100 sign-ups.” If you don’t receive 1,000 visits, was there something wrong with your post? Was it down-voted because it was deemed inappropriate? If you didn’t receive 100 sign-ups, was your messaging wrong on the landing page? Was Hacker News the wrong site? The results likely give you data you can act on.

For the second, you might perform usability testing and measure customer interaction with your product. You don’t want to (and likely can’t) measure everything. Initially measure that which you define as engagement and activation.

Activation is important to measure, since it exposes the gap between when customers first sign up or buy and when they first try to realize the promise made to them. Many problems can inhibit activation during a critical time in the customer-vendor relationship, including password problems, activation e-mails going to spam, shipping problems, poorly understood instructions, misfit, “blank-slate” effect, and so on.

blank-slate effect: The blank-slate effect is the problem of a customer not understanding what to do with a product. In web design, for instance, a user logs in to an application for the first time and it is unclear what to do next to start effectively using the application.

There are many ways to counter these issues, but you don’t want to work around all of them if they all don’t affect you. On the other hand, you can’t afford to miss them if they are occurring! Minimizing the time it takes to get customers going is critical to moving your business forward.

To measure engagement in an early product, you want to measure the specific functionality (features) the customer must use regularly over time to achieve the value proposition. Is there a metric that might establish when the customer is done; in other words, at what point should she have experienced the product’s benefits? In other words, what indicates product satisfaction?

You iterate on product development, messaging and positioning, and market segments until you see organic growth within a defined market segment. This is what the passion metric is for. The passion metric is your own algorithm based on some combination of product engagement data, proclivity to share product, and customer demand for more functionality.

It’s not always possible to directly measure these, but you should substitute non-automated measures where instrumentation is not possible and offline measures where online measures are not available.

Online Engagement

We’ve discussed engagement for online products quite a bit; that’s measuring the specific functionality people actually use consistently over time. Satisfaction with other aspects of your business model include interviews, managing channel programs, customer support, and so on.

Offline Engagement

Don’t let the offline nature of your business dissuade you from using online tools. You are limited only by your creativity in using the Internet to measure offline satisfaction, engagement, and passion.

There are a variety of ways people track engagement of offline products, including loyalty programs, satisfaction surveys, website tools (e.g., support), toll-free product-support numbers, retail-store interactions and observations, third-party reviews, traditional market research, and social media.

Sharing Product

As we’ve stated before, all marketing is based on word-of-mouth. Back in the 1970s, Breck shampoo actively encouraged viral word-of-mouth activity by its customers.² The commercial says: “. . . and then you’ll tell two friends, and they’ll tell two friends, and so on, and so on,” as the screen doubles the number of customer portraits repeating the same words, and doubles again, and again.

Funny how Internet marketers “discovered” viral marketing!

It’s far easier to measure word-of-mouth marketing (including virality) online. Network-effects products are those in which the value of the product increases the more the network of users grown. Theoretically, the value of you using Facebook increases the more your family and friends are on Facebook. (It’s actually not as straightforward as it sounds, since the more people, the higher potential for noise, the less valued the connections, and hence a decreased product value. The quest to improve the experience never ends.)

To measure network-effected word-of-mouth, you measure the number of invites a customer sends out and the number of those invitations that result in new active customers. This is the viral coefficient and must be greater than 1.0 for network-effects businesses that depend on scale to succeed. In other words, for every one person who signs up and uses the product, more than one of their network will also use the product.

If your business model does not depend on scaling to millions of active users (in order to monetize that scale, typically via advertising), your word-of-mouth coefficient does not have to be viral; it does not have to be greater than one! That doesn’t mean, however, you can’t try to achieve that. The higher the viral coefficient, the more organic growth you win.

To induce and measure word-of-mouth marketing for products that are not network-effects businesses, you can do a few of things:

1. Run a campaign that encourages users to invite other users. This can be a temporary friends-get-in-free or a long-term agreement like getting one month free for every new paying customer you bring on board.

2. Use social media tools like Facebook, plus coupon codes to track conversions.

3. Build a method into the product for users to invite friends, and track that as you would the viral coefficient.

Proclivity to share can also be measured through survey-based approximations. Net Promoter Score and MustHaveScore are two examples.

It’s important to understand that product usability is the key to getting users to invite others whether or not you have a network-effects business. People don’t share bad products, they ignore them.

Customer Pull

Urgency for your product can be measured in several ways. Demand for product improvements such as feature requests and even problem reports often indicate a strong need. There’s a possibility for false positives from people trying to be helpful, but for the most part, people who don’t need your product won’t complain about it.

People reporting specific integration requirements, for example, are indicating they are traveling down the purchasing path. Others may voice a desire for different channels, more stores carrying your product, for instance, or specific resellers. These need to be measured not as undifferentiated requests, but rather by market segment. You must be able to discern the difference between demands generating from valuable market segments and those from opportunistic segments, some of whom may take your product or business in the wrong direction.

There are clues in feedback indicating what your “whole product” looks like. The features, channels, environment, and partners your ideal customer segments require to fully realize the value you are delivering is how you expand beyond the minimum viable product.

It bears repeating that building features just because you can is a recipe for missing out on what might make you wildly successful. Also, you don’t necessarily build what is asked for, but rather you first must understand why they’re asking.

Existing Products

All of the preceding applies to existing products, too, but the processes are more complex. You likely already have customers, though it is also likely that you haven’t done the work to divide them into useful buckets. Surveys are a good way to help segment existing customers. Key items to learn:

• What is the core problem they are trying to solve?

• What do they love about the product?

• What would they like to see improved?

• What benefits have they seen from using the product?

• How likely are they to share the product with friends and colleagues? (Net Promoter)

• How do they describe the product to friends and colleagues?

• How disappointed would they be if product were no longer available? (MustHaveScore)

• Where do they “hang out”?

• Who influences their decision making?

Use interviews and engagement data to corroborate survey results. If there are discrepancies, you likely have a problem with survey-question wording.

For complex products and those that have been out in the marketplace for a while, measuring product engagement is more difficult. More functionality means more things to track.

To get beyond this, you must be able to visualize customer product usage similar to the pre-purchase marketing funnel or pipeline: There are a number of actions that a customer typically goes through before purchasing the product, and then there is this magic moment when prospects convert to customers by pulling out their credit cards.

Similarly, there are a number of actions a customer must go through in order to realize your value proposition. Create an engagement map that maps out the core actions a customer must perform when experiencing your product in order to realize value. To each action, there may be a number of sub-steps, a number of features that the customer might interact with in order to accomplish that step. Continue to break down the product to the level at which you can establish the specific product functionality where, if you increase its use, you increase engagement.

This is an example of how one figures out what moves the needle of the business, assuming that increased engagement means increased user satisfaction that might lead to product passion.

For a marketing analytics example, engagement for the chief marketing officer (CMO) might be viewing reports a certain number of times/week. There may be other features, such as communication tools between the CMO and her marketing manager to request new campaigns or reports.

Activation for the marketing manager is creating reports and setting up the tool to track social media, conversion funnels, and marketing campaigns. Engagement might include viewing the dashboards on a daily basis.

The old-school way to improve the product is for internal silos to debate over what features to add:

• Sales wants new features that overcome current objections.

• Marketing wants new features to better position the company against competition.

• Product management wants new features based on customer feedback.

• Engineering wants new features based on new capabilities.

• Support wants specific bugs fixed.

Debate and compromise lead to requirements documents, engineering specification, and a six-month plan for the next version. This method is too slow in today’s world and doesn’t help in determining what the customers actually need.

The high-performing, agile lean startup method means forming a cross-functional team to investigate why one module isn’t performing as expected and then experimenting with ways to improve that specific functionality. The new functionality is tested on a subset of users before being released to all.

Case Study: Instrumenting Growth

Meetup’s story takes place not long after the rise and fall of the Internet, version 1. Only a few years prior, business valuations, Wall Street, home prices, and naïve optimism were soaring. While some were championing the end of business down cycles, others were bemoaning the end of physical books and in-person socialization. The dot-com crash and 9/11 changed all that.

Meetup started in early 2002 in New York in response to 9/11, as a way to bring the community together in the real world, offline.

It was an experiment: Would strangers meet offline after meeting online?

“Remember, there wasn’t social media at the time,” says Andres Glusman, vice president of customer development. “Even the idea of online dating was extremely weird. Would strangers meet online?”

A few months and fifty thousand users later, the experiment looked like a success and the experimental culture that founded the company became a persistent feature.

Meetup started as a free, crowd-sourced, voting-driven mechanism for bringing people together. Only if there were enough RSVPs would the meetup actually happen. Through that system they grew from zero to about a million users. After having discovered a market need, they searched for a revenue model.

They were getting pull from different customer segments.

Andres explains: “The Howard Dean campaign was using it, all of the political campaigns and other large organizations were using it, and they were throwing money at us to build enterprise-driven tools. But that wasn’t really our vision. We were watching how people used the service and saw that individuals were creating their own groups, rather than voting on others.”

So with the million users and money being thrown at them, they made a big bet toward locally driven groups. They re-architected and re-launched the entire service, driven by watching how people were using it. The next year, they risked alienating tens of thousands of customers by shifting from an entirely free platform to one in which organizers paid group dues.

“It was an extremely hard decision,” says Andres. “It was a very difficult thing to go through, but it was another big experiment we made in search of the right business model.”

Meetup lost a great deal of groups, but the activity didn’t go down much. What they were left with was a smaller but much more engaged audience.

As they grew and became more sophisticated, they didn’t move away from experiments, but rather became more efficient at their experimentation. So instead of looking at the data and making big bets, they used the data to “send up trial balloons.”

“It sounds obvious in retrospect,” Andres admits. “There are ways of testing in small ways, prior to redesigning an entire experience, to ensure we’re on the right path. So instead of investing all the time and resources into a big change and then having to support our users, why not invest the time and resources into helping our users learn and evolve as we do.”

All change has a cost associated with it. Not only the engineering time and money, but the implementation and the costs associated with supporting users who must navigate the change.

Meetup has evolved their approach toward launching a bunch of small experiments that inform a big experiment. Once they have insight into something new that is actually working, then they double down and move as deep and as broad as they can, knowing they have a high likelihood of success.

“We’ve put together a usability lab that will run 400–500 usability sessions this year,” says Andres. “Our objective is to shorten the time in between someone in the company thinking, ‘How would a user respond to this?’ and actually watching the user respond to it. Right now, that means never having to wait more than two days to see how a user will respond to something.”

Having achieved impressive user numbers, figured out a revenue model, and implemented fast, agile product experimentation, the key question becomes, what to test? What moves the needle of the business?

Meetup knows the data. They’ve experimented with marketing outreach, corporate users, and advertising, but in the end, it’s all about the meetups. Meetup attendees eventually form their own meetups and then start paying monthly dues.

“The number of people who are interested in attending meetups is provided by RSVP data. You have to join Meetup to RSVP. Fifty percent of all meetup organizers started as meetup attendees. Their product experience is central to them creating new groups inside the platform.”

The product experience requires a delicate balance. Growth can adversely affect value realized by existing customers. This is why feature-mongering is potentially dangerous to a product’s long-term success. Meetup is actually a network of groups. It’s not just about being able to participate in one meetup, but about the ability to serve multiple interests by being a part of lots of different meetups.

“One of the things that we experimented with is to share more about who else is going to the meetups, how you are connected to them, and what other groups they belong to. This very simple test had a huge impact on individuals joining other groups. It proved the concept that we should be optimizing the experience to help people understand more about who else is going to be there.”

To tackle this scope of experimenting, Meetup organizes into five to six cross-functional teams that are focused on what they call an “I-statement.” The teams address a part of the user experience from a user’s point of view. They are trying to solve a problem or enable something really interesting for the user to experience. One of the teams, for example, is an analytics-driven team that focuses on creating smart meetup recommendations. Since enaged users join multiple meetups and engaged users become organizers, it’s important that users get relevant recommendations.

A team is typically comprised of a product manager, a user-interface developer, a back-end developer, a community specialist, and an analyst doing qualitative analysis. Their charge is to design a series of experiments that are meant to make progress toward solving for their core I-statement. They manage a portfolio of different-sized experiments. They are not just satisfied with small experiments that might eke out mild percentage increases in engagement, but insist on going after big changes, too.

Not that the process is perfect.

“When we were running experiments around improving meetup recommendations to users, we ended up changing both how we labeled what we were recommending and the algorithm for producing recommendations,” says Andres. “Problem is, we launched them both at the same time. We saw massive improvement, 40 percent improvement from the change in the algorithm and the change in the header. So at the point we were like, ‘There is something really fascinating that just happened here,’ but we couldn’t point to what exactly and we went back and then systematically undid the experiment we did.

“So we said, ‘Okay, let’s roll back half that. Let’s run a version with just the title changes. Let’s run a version with just the algorithm.’ We ran every permutation we could so that we teased out exactly what caused the jump. We were able to glean this insight into the essence of the algorithm and how people respond to it. We said, ‘Okay, we struck gold here, how else can we use that nugget of insight now in other parts of the experience and propagate it throughout the site?”

What has the impact been on Meetup?

Improving recommendations have led to higher meetup RSVPs, more conversions to paying organizers, and a dramatic increase in growth. But the big impact comes with learning how to ask the right questions.

“One of the reasons we’re experiencing success right now is that we’ve shifted our focus away from thinking about ourselves as groups and shifted to be more about what’s the individual member’s experience inside the context of being part of a group or many groups.

“It’s a subtle difference, but when you come to understand that Meetup creates value as a result of helping you find and engage with the people you want to meet and who are doing the things you want to do, it suddenly is asking the question in a slightly different way. How do we help people find the right people who are doing the things they want to do? Asking that question opened our eyes to all of these different experiments we could run. In running those experiments we saw that there was an impact, that we could make significant impact.

“So what we’ve done over the past year is have an evolution of understanding how we create value for people. We fundamentally believe that great companies do three things really well: (1) ask the right questions, (2) execute many rapid experiments in pursuit of the truth, (3) double down on what they learn from experiments.”

Work to Do

1. Build your actionable data plan.

2. Build an engagement map.

Notes

1. The matter of building data-informed sales and marketing funnels is important enough to merit a standalone chapter. To skip ahead please see Chapter 9: Real Visionaries Have Funnel Vision.

2. See www.youtube.com/watch?v=TgDxWNV4wWY.