How to test two messaging directions with real users

Here's a time tracking app that I downloaded a couple weeks ago for a project that I'm working on. And here are a bunch of other time tracking apps that I either scrolled past, clicked away from, or bounced off the page before getting to that first one. Now, they all pretty much do the same things.

They've got similar features, same basic functionality, same pricing.

So why does this happen? A lot of it comes down to messaging. You land on that page, and you either feel like, " Yep, this is for me. This is what I need," or, " This was made for someone else."

No matter how helpful or useful a feature is, if it isn't communicated in the right way, the people that it was built for might land on that page and think, "This isn't for me," even if it's exactly what they need.

In this playbook, I'm going to walk you through a simple step-by-step process, along with a ready-to-use research template in Lyssna to test your messaging with real users.

So if you're trying to reach a new audience, if you're thinking about testing a new direction with your messaging, or you just want to stop guessing, stop the trial and error and go straight to the source, follow along.

For our example , I'm going to use a fictional company called SkillSpring. This is an online course platform that lets you build and sell your own courses.

A little background. SkillSpring has built a strong following and user base in some of the more, established common categories for online courses: marketing, design, business, web development. But beyond those core categories and audience, SkillSpring sees a bigger opportunity, a growing segment of first-time course creators in areas that aren't as closely associated with online learning.

So think makers, hobbyists, independent educators, people who have real expertise, real skills, who've never really thought about packaging that knowledge into a course.

So the SkillSpring design team has put together a new landing page targeting this audience, and we've got two versions of it.

First is version A. This is our aspirational messaging.

The messaging really appeals to the potential, the opportunity, and I'd say even the excitement of sharing something you love for the first time.

Then we go to version B, which is our more straightforward copy. It leads with clarity, capability, features.

So both of these are aimed at the same audience, and I'll say that both of these are well-written, high-quality landing pages. Both of them could work. But which one actually connects, and more importantly, which one makes SkillSpring users feel like this product was made for them?

That's what we're here to find out, and we're going to start that with our study.

So let's take a look at the study template. Open up yours and you should see what I'm showing here on my screen.

Now, at a high level, this study has three major parts.

We've got a screener survey that makes sure that we're hearing from the right people and getting feedback from our target audience.

That follows up with a preference test. This is where participants are going to see both versions of the landing page messaging and pick the one that speaks to them.

And finally, we've got some follow-up questions. These are open-ended questions to dig into what resonated, what felt off, and what we could improve. Now, this template is built around the SkillSpring example that I'll be walking through, but I designed it so the structure works for just about any messaging test.

So as we get into the details and each section of the study, I'll show you how I built this out for SkillSpring and how you can modify it to test your own messaging, with your own audience for your product.

Let's start with our screener survey. A screener survey is really one of the most effective ways to make sure that you're hearing from the right people in your target audience. And when it comes to messaging, that really makes a big difference.

For this study, I included one simple screener question about each participant's experience level with creating online courses. So in this case, we're screening in people who match our persona and screening out anyone who hasn't considered launching an online course or has way more experience than our target audience.

I've included some more resources on screener surveys that you can dive into, but we've got a simple, effective one set up here.

So let's move on to our preference test

A preference test is a research method that you can use in Lyssna that presents users with a few options and asks them to pick one of them based on your prompt. In this case, participants are going to compare version A with version B.

So both of these are going to be shown side by side, and participants are going to be asked to choose the one that makes them feel most confident about launching an online course.

One tip when creating your own preference test – be specific with your prompt. Don't just ask which option the participant prefers. So for example, if you're comparing dashboard layouts, it's better to ask which one makes it easy to see your most important metrics at a glance.

The more specific that you can make that prompt, the more useful and reliable the feedback will be.

Getting down to our follow-up questions. After participants make their selection in the preference test, they're routed to a couple of open-ended follow-up questions based on the option that they chose. So if they picked version A, they'll see that screenshot again alongside these questions. Same thing for version B.

In these follow-up questions, the screenshot changes based on the choice, but the questions are going to be the same for both groups. First question, what specifically influenced your choice?

Were there any words, phrases, or details that stood out? And that's followed by our second question, was there anything missing or unclear that would make you feel more confident about using this platform?

So the first question tells us what's working, and the feedback from the second question is going to tell us where the gaps and potential opportunities to improve are.

Now that we've got the study set up, let's get some participants in front of it.

Heading over to the Recruitment tab, there are two ways to get participants in your study in Lyssna. First one is a recruitment link. So this generates a link that you can share with anyone.

This is really useful if you have direct access to your own target audience, maybe through an email list or a community, and you want to share it with them directly.

The other option to recruit is the Lyssna panel.

The panel is a great way to get really quick, high-quality feedback for your study. So that's what I'm going to use for our SkillSpring example.

Now, once we click the panel, we've got a couple more decisions. First one is choosing the number of participants. I'm going to start with 20. Now, your sample size for any given study can vary depending on what you're trying to learn, your audience specificity, the confidence level you're going for.

So while there are a lot of factors in choosing your sample size, 20 is going to be a good starting point for this one. And remember, we can always go back and place another order, so don't be scared to start with a smaller sample size and grow from there. Next thing that I'm going to select is the audience criteria.

We've got filters for location, demographics, job titles, industries, interests, and a whole lot more. While we could get really granular with this, I'm going to keep it simple for this study and just recruit participants in the United States, and I'm going to let our screener survey do the heavy lifting.

So this is looking good. We've got our sample size. We've got our audience criteria. I'm going to go ahead and submit the order, and I'll check back when the results are in.

All right. So it's been about 45 minutes, and good news, we've got some responses for our study. So let's take a look and break it down, starting with the preference test.

Here are the results. We've got a pretty clear split and a pretty clear winner.

14 out of our 20 participants chose version B. That was the landing page with the more straightforward, feature-focused copy. Six of our 20 participants chose version A, which had more aspirational-themed messaging. So this is a pretty convincing win for version B, and if we had to pick one of these two directions to go in, that would be the one.

But the head-to-head result is only part of the story. So let's get into the follow-up questions.

I'll drop a link to the full results so you can dig through them on your own, but we'll start with the people who voted for version B.

The big theme in their follow-up questions was that they liked the clarity of it. These participants wanted to understand not just what SkillSpring promises or offers, but how it actually delivers the features, the tools, the specifics to support some of those claims and to evaluate for themselves.

Some of them described the other version, the aspirational messaging, as feeling a little too floaty or being exciting and inspirational, but hard to pin down.

When asked about what was missing from version B, a lot of these participants noted that it just didn't really stick with them. It answered their questions about what SkillSpring is and how it works for new creators, but it didn't make it feel like it was built for someone like them.

Now let's look at the open-ended responses for our six participants who chose version A.

For participants who chose version A, that aspirational feeling that they got and selected wasn't just a nice-to-have. It was a consistent theme throughout that it actually mattered in their decision-making and their confidence to launch a class with SkillSpring.

Words like community, sharing the thing you loved, and even my craft came up repeatedly in these responses. So this version, while not the winner, was doing something that that more straightforward copy wasn't.

When asked what was missing or what would improve that version, these six participants almost always came back to the same thing. They liked the feeling, but they needed a little bit more detail to support how it actually works.

So almost every participant, regardless of what version they chose, was pointing at the same gap, just from a different direction.

On one hand, the aspirational messaging really connected with people. But without something concrete to back that up, it left some questions unanswered. And on the other hand, the messaging around clear features and straightforward copy, this helped people feel more informed, but it didn't speak to them any more than any other course builder site out there.

So now that we've gone through the results and pulled out some of those key insights we've still got one big question to answer. Where do we go from here?

I think the most practical one, the one I would lead with, is to iterate this page and the messaging based on that feedback.

So here are a couple ideas of what I would build on for that next iteration.

First, I would lead with aspiration and back it up with clarity. So I would keep headlines like grow a community around the thing you love, but I would also support it immediately with the tools, features, and examples that make that possible. Next, I'd continue to stick with messaging that makes the first step feel approachable. Both groups, whether they voted for A or B, expressed hesitance around actually launching and even getting started.

So messaging like, "No experience needed. We'll walk you through each step," goes a long way.

And finally, I would simply use the language that participants handed us. We saw multiple responses that highlighted the appeal of sharing a skill or passion rather than just building and selling a course.

When people give you the words and the language that resonates with them, listen, use it, and reflect it back.

So with those updates in place, we could run another study just like this one, this time testing that new version versus our previous winner, and we could use those results to validate that we're moving in the right direction and get any additional feedback that might help us improve it even further.

You have now got a framework for understanding what connects with your audience, not based on assumptions or internal debates, but what your users are actually telling you and how they actually feel when they land on your page.

So if you want your messaging to land, grab the template below, find the right people, and ask them directly. You'll be able to quickly see what's working, and more importantly, what you could improve to build that confidence and that connection even further.

We're putting out more Research Playbooks just like this one to help you better understand your users, make better product decisions, and to feel confident in what you're building and how it's communicated.

So if this one was helpful, like the video, subscribe for more and drop us a comment. If you've got another product challenge or decision that you want to validate, share it with us. Maybe we'll use it for an upcoming Playbook.