KPCC’s new iPad app is approaching its 30th day in the App Store, and its reception so far is encouraging. We’ve had 90%+ 5-star ratings in the App Store, promising engagement metrics, and even some positive press in Nieman Lab.
But it could have gone another way. Every time I ship something new it feels like a hail mary pass, no matter how much I believe in the idea or how well I think we executed.
Ideas are cheap
Before we started designing and building the app, we wanted to establish an audience-informed product and editorial focus. Because of some logistic setbacks earlier in the project, we had to move fast or else risk losing momentum. We knew what our editorial strengths were, but when it came to delivering news on a tablet we were unsure what jobs we could do most effectively for our audience. Being a newsroom in flux complicated matters: we had unified the broadcast and digital parts of our operation, and were producing more text and visual journalism than ever before. We worried, though, that our audience thought of us solely as a radio/audio news brand.
We needed a sign from our audience to point the way, to give us a star to sail our ship by. Since focus groups have well-documented biases towards group-think and social desirability, I knew just asking our customers “what would you like our iPad app to do?” would be unproductive. And because we were short on time, recruiting research subjects and scheduling user interviews wasn’t possible.
Working fast, we established a set of criteria our product validation step had to meet:
- The signal from our audience needed to be an strong indicator of commitment. It wasn’t enough for them to merely say “I like this.” We wanted to create a set of circumstances that emulated the kind of conversion point we’d see when we went to market (i.e., someone reads the product description in the App Store and decides whether to install it).
- We wanted to test a number concepts simultaneously. We had plenty of ideas but lacked clarity, so testing a focused set of concepts to see if there was a clear “winner” was essential to achieving focus.
- We had to arrive at results fast. Given our time constraints, we had to devise a method for validating ideas with our audience that took no more than one week.
The experiment
We settled on running a remote A/B test with a sample population of our current audience, presenting them with four versions of a landing page promoting our upcoming app. The design of each page was identical, but we varied the copywriting, presenting a distinct value proposition and key features for the app.
We crafted value propositions that melded editorial and product strategy, each one promising users a different news experience. We tried to make each concept a distinct and compelling offering. The concepts broke down like this:
- Live listening and breaking news: “The latest news in Southern California and around the world, from voices you trust.”
- All about audio: “The best listening experience for the public radio you love.”
- End of day reading/listening mix: “Catch up on your favorite programs and the day’s news, all in one place.”
- Highly curated: “Breeze through the day’s most interesting news, delivered in smart, hand-picked editions.”
We set about designing landing pages for each of our concepts, taking care to isolate every variable we could. Each landing page got tagged with Google Analytics and contained a different Mailchimp signup form. We then set up a content experiment in GA so that we could evenly distribute traffic among our four landing pages, tracking conversions along the way.
One of the keys to the experiment was presenting it to the audience without them knowing it was an experiment. We used a variety of channels: social media, website invitations, app banners and email marketing. In every case, the messaging promised a sneak preview of our upcoming iPad app, while the landing pages offered to notify users when the app launched if they provided us with their email address. As far as our audience knew, the landing page they were seeing was the only one that existed, and if they liked what they saw, they were giving us their email address in the genuine hope that we would let them know when the app was available.
The outcome
We ran the experiment for seven days, promoting it via different channels in sequence (first by emailing members, then on social media, and through KPCC’s iPhone app).
The results were fascinating. In the first 4 days of the experiment, our “highly curated news editions” concept was far and away the most popular concept, outperforming other concepts nearly 2:1. That held steady until the final three days of the experiment, when we began promoting the experiment to our current iPhone users. To our surprise, our “all about audio” concept began to surge in conversions. By the time we concluded the experiment, our curated news and listening-focused landing pages had proved nearly equal in popularity.
With our concepts tested, we arrived at our product strategy: We led with a twice-daily curated news experience (what became the Short List), also making sure there was a great listening experience there for the users who expected it of us. I’m generally not a fan of splitting the difference when it comes to produce strategy, but providing just enough familiarity has proven helpful when easing KPCC’s audience into our new digital platforms.