Mvp Testing

Testing evolves as you release your MVP and start the trek across the valley of death.

As you first posited and then validated you should have metrics that define customer satisfaction and passion. If you have customers in multiple subsegments, you must determine whether these metrics differ between the groups.

You state as an objective the value for those metrics you must achieve to be ready to scale your startup endeavor.

You run experiments on the product, messaging, and positioning, and acquire (relatively) small tranches of customers to test against and measure the results.

You interact with customers to learn where you are still off on the product, messaging, and acquisition.

For product, measure satisfaction and passion by segment:

Web App Example

Pizza Restaurant Example

You put your MVP into the marketplace and reach out to the early adopters you’ve identified. You measure your satisfaction and passion metrics. If necessary, you run specific campaigns in order to measure these metrics, for example, coupons on Facebook, to measure sharing for the pizza restaurant.

Remember that the objective of “satisfied” is to measure whether you have the right functionality for the right customer profile. The objective of “passion” is that new customers will be put into the top of your funnel with minimal effort on your part, or they’ll pay a high margin, or resubscribe.

Passionate users must be satisfied first and they must take some action to spread the word about your product. If this is not happening, you’re not ready to blow up your customer acquisition. You must first close the word-of-mouth loop. This is a major bottleneck of growth.

At this point you have three primary variables you are playing with:

1. Do I have the right customer?

2. Do I have the right functionality to satisfy the customer?

3. Do I have the right stuff to evoke passion?

If possible, you want to measure in cohorts. What this means is that you are grouping customers based on a shared characteristic—for example, when they first became a customer—and comparing the groups’ metrics relative to that date.

For example, the data looks like this:

Segment A: Passion Data

As a cohort, the data looks like:

Segment A: Passion Cohort

Assuming you made changes to your MVP monthly, you’re able to track the impact of your changes. Note that you also want to compare segments:

Segment-Passion Cohort

It’s important that you’re not measuring everything under the sun. Too much data is not an excuse for not analyzing the appropriate data!

To test, you need customers, though you’re not ready to blow up your acquisition channels. It’s a good time to start testing your acquisition assumptions, though.

In addition to product, you also want to experiment on your messaging and positioning. You’re not in optimization mode here, but rather still learning. The reason this is critical is that messaging and positioning form the promise you are making to the customer. Satisfaction is the measure of fulfilling the promise. Passion is measuring satisfaction plus some other factor that propels users into acquiring more users for you.

In Chapter 3 you hypothesized where your customers hang out, and in viability testing, you worked to invalidate, iterate, and then validate where, in fact, you could locate them. During this process, you actually accumulated people you could go back to and test your MVP with. Although you should continue to work with those people, it’s also a good idea to test with new people based on what you’ve learned.

Using standard methods of low-energy acquisition—reaching out to influencers, running a small AdWords or Facebook ad campaign, using your network, posting on specific forums, tweeting, direct mail—you not only bring in more customers to test, but are testing various acquisition channels.

Once you have customers to test, there are various methods for learning more about your customers. Simple surveys, split-testing (running two or more landing pages and splitting users between them), and online marketing services are all effective depending on what stage of learning you’re in. Such methods really should be augmented through in-person interviews and usability testing, however. Interviews tell you why. Looking at the data tells you what your customer did, but you don’t know why they did what they did until you ask them. Usability testing allows you to watch the customer interact with your product without your influence or the influence of other customers like in traditional focus groups.

As you do when looking for patterns in data, you look for patterns in the customers that you speak to. Can you draw better buckets to group them in? Do they have similar problems using your product? Do they give similar responses to your emotion-evoking, open-ended questions?

Case Study: O2, Telecom at the Speed of the Internet

Lots of organizations say they are customer-centric but what does that mean? Companies have disrupted industries around the idea of customer service itself, so how those companies understand customer-centric must be different from how Google or Apple or Facebook describe it.

Zappos, for example, sells shoes online. “Well, so what,” you might say, “a lot of companies sell shoes online.” Zappo’s shadow force is customer service. From Tony Hsieh’s Delivering Happiness:

“We had a customer e-mail us the other day,” I said. “He had ordered a pair of shoes that we had in our warehouse and we surprised him with a shipping upgrade so that he got his order in two days instead of our original promise of a week. He said he loved our customer service and would tell his friends and family about us. He even said we should one day start a Zappos Airlines.”

Apple, however, offers little to no customer service, yet is arguably more committed to the customer experience than any other high-tech company in the world.

Stephen Devereux was on a team called Futures Innovation at O2, the United Kingdom telecommunications provider. “We were challenging the business, at that time, to really think differently about how we delivered products and services to our customers. We deliberately branded ourselves differently from the traditional sort of strategy name to be more forward thinking, and deliberately use the word ‘innovation’.”

As a telecom company in an industry not known for being either customer-centric or innovative, O2 had performed well, not only moving its way to the number-one telecom provider in the U.K., but also building a brand recognized and admired around the world. At the same time, they could see the signpost ahead: being the plumbers of the Internet had run its course. Of course, many telecoms around the world have realized this at some point or another. U.S. companies seek government legislation to thwart innovation and protect their existing oligopolies, while squeezing every last penny out of intermediate technologies like ISDN and DSL. They spend hundreds of millions on high-priced consultants trying to figure out ways to carve out new revenue opportunities, such as applications or GPS services.

Yet disrupted they are.

O2 had a key insight, however. Devereux says: “We were in a very mature market, and we saw that our new set of competitors were not the telecommunications companies of the world, but the Internet players of the world: Apple, Google, Facebook. We had to move at Internet-speed, Twitter-speed.”

These companies are constantly releasing new product. They are in beta for years. Updates come weekly, if not daily. They are constantly running experiments, optimizing product delivery. “We were lucky to deliver something quarterly. We were challenging the business to start working in a completely different way to try to take these guys on and compete with them.”

The thing is, being customer-centric in this way is completely different than the customer-centricity that elevated O2 to number one in their market. “We have to think of the customer” becomes a proxy for brand protection, rather than innovation. The organization’s size and complexity—people responsible for customer service, retail, PR, brand, all various voices of the customer—created obstacles to new products. “To start off any new product in our business, you have a cast of hundreds of people, because everybody has their bit of say. We were actually creating a catch-22 situation for ourselves in which we became full of inertia, unable to serve customers with new products, because we were so worried about the customers’ experience,” Devereux says.

The key to change was to move our more innovative endeavors to a sub-brand, where a team had direct access to experimenting with customers. In the case of O2, the sub-brand was a philanthropic initiative called Think Big, started by CEO Ronan Dunne, who believes that being part of a large organization means having the corporate responsibility to have a positive impact on society, to protect the environment, and to help the youth of today. Releasing products under this sub-brand, far away from the traditional telecom services, had a number of positive implications:

• Initiatives would not be subject to the same financial measurements as regular products.

• Customer expectations would be different. It allowed the company to experiment with internal changes.

The first product released was O2 Learn. The vision for O2 Learn was to provide a place online for students to find a video about a school lesson—like to get a bit of history on Lord Nelson, for example. A small cross-functional team consisting of a product manager, a designer, and a few engineers worked to release a version quickly. Devereux explains: “The user experience and the interface were prominently designed around students needs. Secondary to that was what teachers needed. We thought quite hard about how teachers could easily upload videos, but the rest was around the students.

“What we found very early on was that teachers were using the site as much as students. What we found was that teachers wanted to find out what good lessons look like so we immediately worked to improve the site for teachers. If that had happened in a traditional way then project managers and designers would say, ‘Oh gosh, that’s a change in the requirements. You have to go back to the beginning.’ You know the process.”

The power of the O2 brand meant that after releasing to a small group of users and ironing out the kinks, it was quickly in front of thousands.

“We had a quick time slot in order to learn, but then there was this chap, a famous celebrity chef in the U.K. named Jamie Oliver who was teaching school dinner ladies how to cook better foods for kids within their budget constraints and we ran an ad for O2 Learn on the show. So it was a really nice alignment there. As a result we saw massive spikes and it just started to grow from there. We started actively promoting last year and started doing these challenges every quarter where we’d give money to the best lessons.”

Although not particularly disruptive technologically, O2 Learn made a difference to tens of thousands of kids and was clearly disruptive inside O2. “It got massive airtime within O2. This one and a couple of other projects were instrumental in proving the case that actually there is a different way of doing things. We were able create a lab and we got headcount of nearly 30 people.”

It’s a big win for O2, as well. O2, already known as a leading telecom, and a world-renowned brand, is recognized for assisting teachers and students. These are the sort of initiatives that, for large companies, move customers from satisfied to passionate.

Work to Do

In the MVP phase, the key metrics are user satisfaction and user passion. That’s it. If you know your customers well, you should be able to report these two metrics by product functionality, by customer profile.

Notes

1. If not congruent, you are ripe for a pivot.

2. See http://news.ycombinator.com/item?id=542768.

3. Our friend Sean Ammirati suggests, somewhat tongue-in-cheek, that some entrepreneurs can be helped by telling them to build MAPs (Minimally Awesome Products) instead of MVPs. This is the difference between calling a car “used” or “certified pre-owned.”

4. www.forbes.com/sites/brianclark/2012/05/23/audience-lean-startup.

5. www.fastcompany.com/1837168/pop-goes-pivot.

6. For an amazing read on the subject, take a look at Ryan Holiday’s Trust Me, I’m Lying: Confessions of a Media Manipulator (Portfolio, 2012).

Chapter 9

Real Visionaries Have Funnel Vision

All you need to know about business today is that the Internet changed everything, right? When it comes to distribution, sales, marketing, the Internet changed everything!

Didn’t it?

If you say yes, you’re likely viewing the world from a seller’s perspective. The activities and tactics you employ to make people aware of the value you hope to provide them, convince them you’re capable of providing it, and make them trust you with the ability to get it done change over time. But that you have to go through those specific steps doesn’t.

What motivates individuals to proceed along the path from awareness to trusting to passionate varies by many factors, including market segment, product, and emotional pull. This is true whether you’re selling multifunctional printers to businesses or social apps to consumers. In a similar vein to “if you’re committed to solving a specific problem, you should be solution agnostic,” once you have targeted a specific market segment, you should be agnostic toward the sales, marketing, and distribution tactics required to serve that segment.

Beware the business “canvass” you’re filling out. The more disruptive your startup endeavor, the less you know about your marketing and sales funnels, despite what you think. Just as with product development, a lean startup must learn how to market and sell.

Entrepreneurs without funnel vision conflate the two sides of a transaction: how the buyer desires to buy and how the seller desires to sell. What results is likely a horrible mismatch. The business executes a process they are familiar with, rather than the one that their buyer implicitly demands for a particular product. The buyer will not engage with a sales process that doesn’t fit with what feels right to them. You can have an incredible product capable of delivering real value that never sees the light of day because of the wrong sales and marketing practices.¹

Founders often view salespeople as experts in convincing potential customers to buy. They view marketers as people who run campaigns and build brand, and generate buzz, in order to fill the top of the sales funnel; to find potential customers to sell to. This view is fraught with problems, not the least of which is that sales and marketing are viewed merely as activities that need to be executed. If the execution fails, you need to execute the executors. If nothing else, this is a recipe for high staff turnover, departmental goals that diverge from company objectives, and sales to anti-segments.

No one truly wishes to be convinced to buy or to be sold to. If an individual is at all inclined to purchase a product, they wish to be reassured that the product will provide them the value they expect. The buyer’s process is to learn as much as they can about the product, test the value proposition if possible, evaluate the company, weigh the potential value versus cost plus other factors, and then make a decision.

The Internet has changed none of that.

The selling process should, therefore, be comprised of the activities necessary to provide the buyers what they desire in order to move them through their buying process. Activities might include building a web landing page, crafting messaging, creating a brochure, attending a tradeshow, making a phone call, and so forth, depending on your product and segment. There are an infinite number of activities, which is why it’s absurd to think that you know what the best activities are for your particular product and your particular customer profile. Yep, the Internet changed everything.

We once asked the winner of a Southern California university business-plan competition what he planned on doing with his $25,000 award, and he said, “I don’t know—build a website, get some business cards . . . .” Really?

For some reason, an incubator we know was working with a graphics design company and a few marketing professionals to create a super-high-level marketing plan template for startups. Unsurprisingly, when it came time to develop the plan’s list of first things to do, the list included design a logo, develop a website, print letterhead and business cards, buy pens with the logo on it, and so on.

Really?

Come on. You’re not truly in business until you’ve got a business card that says so. And a website. The Internet did change everything!

Entrepreneurs are quick to either:

• Adopt the hippest marketing trends of the day. (How many articles entitled “Why You Must Have a Social Media Strategy or Fail!” must we endure?)

• Immediately hire an agency because the founders know nothing about marketing. (What’s the quickest way to burn your cash? Hire a PR agency to manage your must-have social media strategy.)

Take a deep breath.

Step back from the Twitter. Strike that. Step all the way back from the Internet.

If you operate in a highly uncertain environment—in other words, are toward the disruption end of the innovation spectrum—reject best practices. Best practices are for standardizing and optimizing the execution of processes known to work. Best practices are antithetical to disruptive startups. Experts apply best practices based on accumulated knowledge that may or may not apply to your product and your target market. These experts are unlikely to be experts in the domain that matters most to your business.