Chapter 6
Viability Experiments
Winners experiment. The world is changing fast—strike that, the rate of change is increasing. You are being disrupted. The pundits screaming at you from the TV, what are they on about? They’re being disrupted. It’s what you do when you become obsolete. Innovate or die. You can double down on “ignorance is bliss” or you can start running experiments.
The idea of viability experiments is to test critical aspects of your business model, before you build them out. Just as a scientist runs experiments to prove a theory, so should an entrepreneur. Just as researchers and engineers use iterative processes to discover new things, so should founders use such processes to discover what will work in the market.
“The real measure of success is the number of experiments that can be crowded into 24 hours.”
—Thomas Alva Edison
As quickly as you are able to posit your market segments, you want to think of ways to validate that the segments exist as you describe them. Getting out of the building to interview prospective customers gives you your first insights into whether you’ve described them and their problems correctly, and now by running experiments you work to confirm customer behavior. Experiments represent the first opportunity for your potential customers to experience at some level your value proposition.
Augment what customers say they’re going to do with tests that show what they actually will do!
The basic premise of viability experiments is that the quicker you can determine a failed approach, the better. If you require an expensive custom mold to be built for your product, wouldn’t it be better to understand the viability of the product prior to building the mold? If you wish to open a restaurant, shouldn’t you check whether people like the type of food you plan on serving? If you must develop software for a year to finish a product, isn’t there a way to test whether you’re building a product people want?
Viability experiments do not only apply to product. You should test any element of your business model that has significant uncertainty. Will a retail store carry a product like yours? If so, what do you have to do to get shelf space? How many page views do you need before advertisers will pay you enough money to sustain your business? Will people buy high-end art online without experiencing the piece in person?
Eric Ries adopted the term minimum viable product (MVP) to describe these experiments, as long as they result in validated learning.1 Originally the phrase described the minimum functionality necessary to be viable in the product, as evidenced by the willingness of customers to pay, because it sufficiently addresses their problem or passion.
Ries’s definition includes all sorts of creative viability experiments that result in learning and reduced business-model risk. We believe it’s valuable to separate out the first attempt at putting product in front of customers and call that an MVP, versus running purpose-built experiments to educate your product development assumptions and other risky aspects of your business model.
So for this book, viability experiment refers to tests you run to validate aspects of your business model. This epitomizes Ries’s build-measure-learn framework. We are holding out the term MVP for the first stab at releasing product that creates value for a specified customer segment. What follows are the most familiar methods of running viability experiments.
The Infamous Landing Page
The landing page2 test is probably the most familiar to people. The basic premise is that you put up a basic one-page website comprised of your messaging and a call to action, drive traffic to the page, and measure clicks on the call to action.
Landing Page smoke tests are at the same time the most abused and underutilized of tests. Most abused because the interpretation of the results is highly subject to experimental bias: Entrepreneurs see what they want to see. The Landing Page test has three variables in play:
1. How good is your acquisition method? If you drive traffic using Google AdWords, but nobody is searching for your solution, then you haven’t really tested your value proposition but, rather, your ability to drive traffic via search marketing. Furthermore, if you don’t draw the right traffic, in other words, your target segment, then you can end up with hundreds of e-mail addresses from casual passers by who have no deep interest in your product.
2. How good are your messaging, positioning, and design? People who click on the call to action are persuaded (among other things) by their belief in whether the product is targeting them (positioning), if it might benefit them (messaging), and if the design reinforces the product appeal to the necessary degree. If you’re selling bridal products, but the landing page looks like a GeoCities website circa 1999, you’re going to have a hard time converting visitors even if they would value your product.3
3. How good is your value proposition? If you’ve driven the right people to the page, have articulated your messaging and positioning well, and your design doesn’t subtract or distract, then you might, in fact, be able to test your value proposition. If people don’t click on the call to action, you have a long road in front of you. If people do click on the call to action, you still have a long road in front of you, but perhaps enough reason to continue the journey.
At this point, you might reasonably ask, why run a Landing Page test? Here are some good reasons:
As mentioned earlier, if you run a high-quality test, you can learn whether you have a fighting chance.
Collecting e-mail addresses provides you a list of people to follow up with as you pursue the idea (including, one would hope, early adopters).
There is the possibility, again, of truly validating an idea.
Recall that Lucas Carlson at AppFog started with a landing page test that generated 800 e-mail addresses overnight. Unlike other entrepreneurs who instantly believe that whatever number of addresses collected proves idea validity, Carlson doubled down on testing with what we call a “high hurdle” test. He wanted to only let into his beta program early adopters willing to jump through a hoop. The willingness of customers to fill out his painfully in-depth survey was great validation of the idea through indication of a deep level of pain.
How, then, is the landing page underutilized? Because it’s still primarily a tool for high-tech startups, when in fact it could quite easily be deployed by startups outside of tech, as well as by big companies.
We once counseled a biotech entrepreneur who had invented a technology for folding proteins. He related to us that he was stymied because his technology appeared applicable to many, if not all, protein families with some future investment and fine-tuning. However, at the time, it only worked for a specific set of protein families. As such, he was unsure whether the protein families he could fold at the time were commercially interesting, and whether he should invest time and money into configuring his technology for another specific set.
We reframed this question through the lens of market risk and technical risk: Was the current set of proteins commercially viable? Yes or no? What was the technical risk of supporting other protein families?
Either way, what sets of protein families, irrespective of his ability to fold them, showed market pull? In other words, could he de-risk the market further before accumulating additional technical risk?
We suggested he consider a landing page viability test that allowed those working with specific proteins to opt in based on information provided. Then, we also asked him to consider running ads in a medical journal pertinent to his target market segment, which would actively test demand for various, specific protein families.
Concierge Test
The Concierge test is testing by doing. Never a week goes by that we don’t hear of a new online multisided marketplace idea. These entrepreneurs have grand plans to build an online platform that brings together two or more parties who aren’t readily accessible to each other.
In the instance of creating a new market, in other words bringing new buyers and sellers together, the Concierge test is conducted by manually bringing buyers and sellers together and repeating that process until you learn (1) whether taking a cut of the transaction would support a business, and (2) what part of the transaction can be automated.
This test might be useful in non-marketplace examples, too. In many instances, if you can’t make a transaction happen in the real world in real time, it’s not likely to work online, either. If nothing else, you will learn what it will take to get the parties to the market.
Services-based businesses are good examples of Concierge tests. If someone pays you for a service, you pretty much know a pain exists and someone is willing to pay for it. (This is beyond the point that many product startups get to!) This, of course, doesn’t mean you have a scalable business. Whether a product-based startup can be built from a successfully sold service is no small matter.
Converting a services business into a (scalable) products business is always a tough proposition.
(We once worked with a successful consulting business to help establish a products division. Despite the fact that executives could never get their heads around non-billable hours, we were able to build an MVP, execute one major pivot, and land a six-figure annual license deal within eight months. Because of not meeting business-plan expectations and other internal issues, however, even this win was not enough to save the project.)
Presuming one is able to prove viability with the services version, what is the next biggest risk to offering a product-based solution instead? Two things come to mind:
1. Is the problem even solvable with a product? Believe it or not, some problems are not. Some problems are human based. Problems caused by personality, habit, emotions, or behavior may be big problems, but products might not do the trick. (Products relying on infomercials and smarmy direct-marketing techniques often rely on this fact in order to sell products to the unaware.)
2. Are customers willing to substitute human service with a product? In long-term services businesses, the trust engendered by personal relationships is often necessary to create passion in the business. This may be threatened by product. Which isn’t to say that you shouldn’t pursue the opportunity, but you might need to look for a different market segment.
To move toward a product-based solution, one needs to find what within their existing process can be automated, using existing passionate, service-based customers as early adopters. They will be your bellwether to see if you can provide benefits of automation beyond the value your service-product offers.
Case Study: Curating User Experience
Steven Cox’s vision was to let musicians teach. Through his own experiences as well as those of friends, he knew that the toughest part about being a musician is making a living. So many musicians turn to teaching music. The toughest part about teaching is finding and retaining students. So being an Internet entrepreneur as well as a musician, he decided to combine those passions and create a lead-generation site for music teachers, called Click for Lessons. We spoke with Steven and his co-founder, Chris Waldron.
Cox experimented with several MVPs that alternatively offered unlimited leads to teachers at $30/month, tried to create a free model with advertisements, and sold leads on a per-lead basis. It was an idea—bringing music teachers and students together—in search of a business model. They grew the lead program to over 4,000 instructors in 15 major U.S. cities with revenue exceeding six figures, but growth was slow and hard-fought, and eventually stagnant.
Through failed experiments and speaking with teachers, the Click for Lessons team learned a couple of key points:
The musicians they targeted were living paycheck to paycheck, and many didn’t have credit cards.
If they paid $6/lead, they fell $6 in the hole.
To find students, Click for Lessons simply used online marketing techniques to snare students looking for teachers, and sold their information to the handful of teachers they had recruited.
Steve Cox: “The epiphany came after talking to a bunch of our instructors. I’m telling them they can earn thousands of dollars per lead—they just have to get them to buy-in. I’m frustrated, complaining to a buddy, who says: ‘The problem is you’re trying to get the musicians to be like you.’ And I thought, ‘He’s right.’ I’m a business guy; I like the interaction with my customer. These musicians don’t really want to run a business. At the same time, we’re getting angry calls from the students, saying, ‘You guys are worthless—none of the teachers called me back.’”
Cox went back to the office and pulled the plug. Several days later, they had a new MVP.
Chris Waldron: “It was January 2008, and based on what we had learned we went out and recruited 30 instructors by hand. We got them to sign agreements and commit to certain revenue or pay rate that we were willing to pay them, and then we hand matched them to actual students. For the students, we rigged together a phone system, literally like ripped open the wires and rewired it. We had some open source from a phone system, and we hacked together a call center and published an 800 number and basically for half a day took calls on the student side and the other half recruited teachers.”
A classic concierge MVP!
Up to that point, Click for Lessons was just an Internet marketing play, which wasn’t the type of business they wanted to build. It matched neither their values nor their vision.
Cox: “If all we could be is a company that was really good at search engine optimization, that wasn’t exciting to us. Music is emotional, and by sitting in the middle of the transaction we have all the data related to the actual student experience. That was exciting for us to be able to leverage in building an actual brand that could be considered the place to go to buy services. The vision that we started to create in 2008 is to become the Amazon of services.”
So they had finally identified a viable online marketplace, but the Internet is awash in failed marketplaces. Do a Google search for any local small business and you will get a dozen online business directories, each imploring owners to “claim their business,” each with pedestrian five-star consumer reviews capability. To grow, Click for Lessons needed to establish its “shadow force”: its core competency not readily visible to the outside world, but that drives market differentiation. They rebranded as TakeLessons and learned to bring new value to the equation.
Cox: “We continue to evolve the platform as we figure it out. We didn’t know what was required, but we knew that we had to control the experience more than before.”
TakeLessons learned that if they can deliver enough students to the teachers they’re okay with taking a lower price as long as they’re making consistent money on a weekly basis. A typical marketplace brings buyer and seller together and then leaves it up to them to conduct the transaction, but that’s not the way small business works.
Waldron: “One of the biggest insights when moving to this model, which still blows my mind to this day, is there are many teachers who would rather not address the fact that their customer owes them money. They’re concerned they’ll lose the customer. They felt awkward in front of little Johnny saying, ‘Hey, tell your mom to pay me $150 this month,’ and they would teach for another month without getting paid. So we saw that if we collected the money, the teachers feel more comfortable.”
Cox: “It’s like we were getting back to the vision the long way about. Teachers just want to teach. They don’t want to market, sell, schedule, or collect payment. As a platform, we needed to provide value so the teachers want to be on the platform.”
Unlike other service-based marketplace platforms, which must bend over backwards to keep teachers from violating terms of service by going outside the system, TakeLessons was creating demand to be on the platform. The big win for the teachers, of course, is steadier work and peace of mind.
So hiring the next instructor in New York City on the same block was not as valuable as hiring someone in the next market over. TakeLessons looks at the demand in a particular area: all the leads, all the searches, and all the visits, so they’re able to identify hot spots in the system.
Waldron: “What we’re able to do is to understand that if this teacher application comes through and she says that she travels five miles and also has a studio, what we’re able to do is to create what we call a coverage score. So this is 100 percent coverage, and then based on the subjects that she teaches and based on whether she travels, we’re able to see how much of our demand she covers.”
They know when they add a teacher to the system they are getting a return on marketing dollars. They can also see from a national perspective where demand is short. To make this work, of course, they needed students, too, but rather than just leads, they needed relationships. Building out the call center rather than driving bookings online allows TakeLessons to not only increase the conversion rate, but learn. Many companies are so focused on automating and reducing costs, they don’t have the opportunity to talk to and glean insights from customers.
Waldron: “Our revenues would be literally a tenth of what they are if we did not have that high-touch method. I can tell you that in our student support call center we average 175 calls a day, Monday through Friday. One thing we found interesting, for example, is that one reason why a student likes going with us is they actually don’t feel guilty when they cancel. They can call up TakeLessons, cancel with that teacher, get another teacher and a 30-second conversation and there’s no guilt associated with quitting.”
Cox: “The key insight we had was the need to protect the student’s experience.”
In order to protect it, TakeLessons disengages with underperforming teachers on a regular basis. If they find the instructor routinely gives a less-than-stellar student experience, TakeLessons chooses to place the student with another teacher.
The student’s experience is largely dependent on the quality and dedication of the instructor. The experience is made better by curating instructors.
Waldron: “When we talk about our teachers, it’s not that we scraped 30,000 profiles and called these our teachers. Our teachers have signed contracts, they’ve attended orientations, and they have agreed to uphold the way we treat customers.”
Student complaints obviously affect teacher evaluations, but data also helps inform decisions. TakeLessons monitors student engagement by subject, billing plan, geographic area, and by instructor. They measure lessons taken per month to indicate that they are learning and progressing. They measure month on month retention by calendar and user cohorts.
Teacher engagement is measured by many factors, including how often they update their calendar.
Waldron: “We could sign up a hundred instructors this month but if none of them log in 30 days later to review and update their calendar, the teacher’s never going to respond to bookings.”
They measure the time to contact new students.
Waldron: “So when we send them an alert either by text message or by e-mail saying, ‘We have a new student for you,’ we measure the next time they view their dashboard. We can see how quick they are to reply and respond. Once they actually talk to the student they have to click a link and we measure that. We can align or measure how closely related that is to the teacher’s retention and how good they are and the quality of the students’ experience.”
TakeLessons does webinars so teachers can learn from other teachers, and they measure attendance.
Waldron: “We have a teacher in Atlanta whose average retention is 40 lessons, which is amazing; so how does she do that? She’ll put together content and we facilitate a webinar and she’ll run it, and teachers from around the country follow up after with questions.”
Having grown up with a relatively high customer acquisition cost (which enabled their learning), much of TakeLessons analytics focuses on reducing that cost and maximizing lifetime value. So they run experiments.
Waldron: “We often test pricing. We’ll take a teacher, for example, say Jessica, she’s in Houston and her average retention is 22 lessons, and we have 60 students to book. So we’ll adjust her pricing maybe up $5 and down $5 and run a cohort analysis of students who are starting at the same time, with the same teacher under different pricing, and then see how that relates to their retention. The goal is to maximize the lifetime value. So if we charge $5 less but the student stays 14 lessons longer, we’re making more profit on a per-student basis, and the teacher is happier as well.”
It’s an intricate loop whereby TakeLessons works to help teachers be better teachers, in order to protect the student’s experience, in order to retain students so that teachers are more successful. This loop is TakeLessons’ “shadow force.”
By constantly working to improve the loop, they increase customer passion on both sides of the marketplace, grow, and differentiate themselves from the competition.
Cox: “Our next big inflection is where we figure out how we can get students to share their experiences, and the music teacher community is very connected. So how do we continue to provide more value to them? We need to continue to help instructors learn from each other how to be great music teachers.”
Waldron: “So that’s the point, do we have product market fit for delivering on the one thing that both teachers and students want, which is each other? I would say yes, and we do that job excellently, but in order to build the vision we need to create more passion. We talk about it, and Steven just reinforces that all the time.”
Cox: “It’s like there is no product-market fit. We have to get more teachers where they’re needed and make them better every day. We need to improve the user experience every day. We need to continuously add more value.”
Waldron: “That’s why it’s really hard if you want to build a brand that people trust. We could just create a local dot-com or a Yelp where it’s just fine, but our vision is to be able to take this and expand it into other services that people buy for their home. If we want to be able to do that and then leverage that, we have to build a brand people trust.”
Cox: “Our investor, Maynard Webb (former COO of eBay), told us the next generation of marketplaces like TakeLessons were building a marketplace for the equivalent of the eBay Powersellers—proven merchants with good track records—something the company can stand behind.”
Wizard of Oz Test
Similar to the Concierge test, the Wizard of Oz test requires little or no actual product development. It could also be called the Hamster Power Generator test, similar to the cartoons in which you discover hamsters on exercise wheels are powering the car engine. Basically, you provide a front end for your product where someone can buy, yet you fulfill the product’s purpose manually.
This is about as lean as it gets. In lean manufacturing, a production stage is started only when demanded by the downstream process. In the Wizard of Oz test, the product itself is in action only when requested by the customer through placing an order.
Zappos started out as a Wizard of Oz test.
CEO Tony Hsieh tells of meeting Nick Swinmurn, founder at Zappos, in 1998:
Nick summarized his entire pitch in three sentences: “Footwear is a $40 billion industry in the United States, of which catalog sales make up $2 billion. It’s likely that e-commerce will continue to grow. And it is likely that people will continue to wear shoes in the foreseeable future.”
He reserved the domain name Shoesite4 and without building out any sort of inventory system, he literally walked down the street to the local shoe store, took photographs of their shoes, and posted them on the website. Several months into the project, Nick was getting $2,000 worth of orders per week. They weren’t making any money, because after an order was placed, “Nick would run to the local shoe store, buy the item, and then ship it out to the customer.”5
In 1998, Nick Swinmurn used a classic example of a Wizard of Oz test to test market risk. So did another company from the same time period, CarsDirect.
Case Study: Idea to Wizard of Oz in under 90 Days
Day 1
Back in 1998, Idealab founder Bill Gross wanted to buy a new car. So off he went to his local Southern California dealership, where he immediately experienced the typical new-car-lot runaround: “Oh, that car’s not in stock. What you’re asking for is very rare. We don’t have that color on the West Coast. You’re better off buying what we have on the lot,” and so on.
Like the entrepreneur he is, Gross said, “There’s got to be a better way.”
He decided to build a website where you could go online, configure the car you want, see the price right then and there, put a deposit down with your credit card and you’re set for delivery. Big deal, you might say, but remember, this is 1998. Amazon.com was three years old. The idea scratched Gross’s itch, but lots of people pushed back on the idea:
“People won’t use their credit cards online.”
“No one will put money down on a car, sight unseen.”
“Maybe you’d do that, but there’s no way anyone else will.”
Rather than sit around and debate, Gross built a test. “Let’s put up a website and let’s build the proposition out of what we want to offer, and then let’s see if anybody wants to do it; if so we’ll build
the company, and if not, we won’t build the company, but let’s just find out.”
Idealab hired a founding executive, “Bob,” to help prototype the idea within 90 days.
Day 45
Bill asks Bob how it’s coming along and he says: “I’ve been calling all the car dealerships, Ford and Honda, and I’m having a difficult time sourcing the cars.”
Bill responds, “What? What are you calling them for? We’re just going to put this up and see if anybody wants to buy a car. If someone buys, we’re going to go down to the Honda Auto Mall and buy the car at retail and deliver it to the person on a flatbed truck and lose money on it.”
Day 70
Bob has a problem building the “configurator.”
“The what?” Bill asks.
“You know, the way to allow the customer to choose all the different options. It’s really hard, too many packages, too many different variables to make this work cleanly.”
“No, no, no. Just let them type in a blank field. We’ll have a human read the thing. Don’t worry about it.”
Day 80
Bob has run into another issue.
“Launch the site,” Bill says.
“You haven’t even heard . . . ”
“Turn it on!” So on a Thursday night, Bob launches the site. Friday morning, Bill meets with Bob.
“Well, what happened last night?” Bill asks.
“We sold four cars.”
“Hurry up and turn the site off!” They quickly took the site down because they had just lost money on four cars, not just two, but at least they had an answer. People will order cars sight unseen on the Internet via a direct-sales model.
CarsDirect became Internet Brands in 2005 and went public in 2007.