Workshop transcript

Name: Live workshop: Screener survey mistakes (and how to fix them)
Start: 2025-03-12T00:00:00.000Z
End: 2025-03-12T00:00:00.000Z

Question

Workshop transcript

Accepted Answer

Let's get started. So quick intro my name is Joe Formica. I'm the design advocate here at Lyssna

Basically, I've been a senior product designer researcher for 10 plus years, and I've been teaching stuff related to product design and UX research for almost as long. And in my time working in. Product design research. I've worked on lots and lots of projects, done lots of screener, surveys, made a lot of mistakes with screener, surveys early on.

As with all workshops, I'm gonna be bringing that experience in, bringing in some examples, and hopefully helping you learn from some of the things I learned the hard way years ago. You can skip that part and do it the right way. Going right into your research and the stuff that you're working on.

In terms of this workshop and just about any workshop that I'm teaching, my goal is for you guys to leave here today with some real tangible skills and some confidence to improve your day-to-day work. I used to, and still do, attend a lot of panels, webinars, stuff like this. And I remember, early on in my career as a product designer and researcher, there would be a lot of these, I don't know, meetups, panels, webinars, et cetera, that I would leave feeling like I know more. I have more information more stuff stored in my brain, but wasn't really sure how to put it into action or didn't really know what to do with it. Let's say when I came back to the stuff that I was working on every day, and I think there's some room for that for sure, but the way I like to teach and the stuff I wanna bring you is practical.

Giving you that confidence and really hopefully making those connections to the research that you're doing, the stuff that you're working on, and giving you that confidence to put it into action. Today, tomorrow whenever it comes on your plate. So the two big questions that I'll ask you at the end of the workshop to see how I did today. Number one, did you enjoy the workshop? You guys are taking an hour outta your day to come sit and listen to me talk about screener surveys. I wanna make sure it's as enjoyable, comfortable as possible. And more importantly, do you feel like a better, more confident designer, researcher, product person after leaving this than you did, an hour ago at this point? So I'll ask those at the end see how we did. But that's me. And that's what I create all these workshops for and create them with the goal of hoping that you get out of each one of them. All right, so some quick tips for getting the most out of our workshop. Like I mentioned, everything is recorded, so you guys are going to get the recording of this workshop.

I put the slides in the chat, I'll share those as well. But everything from the workshop is either already here or I'll share with you during or after the workshops. You'll be able to come back to that if you have to hop off early. If you gotta leave for a couple minutes to put out a fire and answer, answer some emails, you'll have all this material in your inbox so you can come back and check it out, review this stuff and use it when you're going to do your own screen or surveys.

Second thing, we're gonna keep the chat open and active. You guys have already done a great job of that. If you have questions, especially since we have relatively small group here today, drop your questions in the chat. I'm gonna try to get to them as many questions as I can. And put 'em in when you have questions.

If I can't get to them immediately, I'll mark them, put them in the q and a section, and hopefully find a minute or two to either have a little checkpoint or a little break where I can answer them. I'm also gonna be asking you guys some questions, thrown out, some discussions out to the group, some little mini activities that we're gonna be doing.

So that'll all happen through the chat. So keep it open. Don't be shy to contribute, stay comfortable, hydrated, and happy. We're only here for an hour, but, make sure you got some water, make sure you're good. Hopefully in a quiet, prone to con concentration environment. And of course, if there's anything I can do to make it more comfortable, make a better experience for you here today.

Just lemme know. All right. Our last setup thing before we get going here is a little fig jam template that I created for you guys. We're gonna be using this throughout the workshop today. I'm gonna direct you back to it now and again for a couple quick exercises that we're gonna be doing. But I think more importantly, it's gonna be something for you guys to carry after the workshop and refer back to as something you can use when you're creating your own screen or surveys.

So go ahead and click that link that I just put in the chat, and you should see what's up here on my screen. This is the companion template to this workshop. If you have this open just click this button. If you're logged into Figma, open in Fig Jam, do it to my team here. And that will create hopefully yes, a duplicate your own copy of this.

Fit Jam template. I'll go through this and I'll walk through each of the steps. I know there's a lot here, but for right now, just make a copy of it and keep it open. We'll be referring back to this later. Basically it's a template and it's gonna have everything from the workshop today along with some other resources, some spaces for activity and stuff like that.

So we'll be covering, I'll walk you through everything, but for now, just hit that link in the chat. Click open in fig Jam and keep that copy open. We'll come back to it in a little bit. Let's see. I'm gonna give everyone a minute to do that. I see we got 19 people who did. If you're not logged into Figma, if you don't have that open, if you can't do it right now, no problem.

I'll be sending this out later as well. Can still totally follow along with the workshop but it will help if you have it both for during the workshop and after. Looks like we got 20 people. Nice. Cool. If anyone has any issues with that, let me know and I can help you get that sorted out as well.

All right, cool. Lemme check that one more time, make sure we're all good here. Nice. Okay. Awesome. All right, so let's get into what you guys are all here for today. We're talking about screener surveys. We're gonna start with a little intro here, and then we're gonna get into the title of the workshop, screener, survey Mistakes, and more importantly, how to fix Them, how to correct Them, and how to drastically improve not only your recruitment, but your research in general for whatever your next study is.

Let's start out though with a little overview here, so screener, surveys, I guess I could even back up a step further, but. Screener surveys are a couple questions that we're asking as a means of recruiting people for a certain research initiative. We're gonna cover a, an example throughout today's workshop and improve that screener survey.

But the idea is that we wanna run some usability tests. We wanna run some interviews. We need people, we need participants to do that. And most likely we want participants who fit certain criteria. If we're looking to create an app for parent or website for parents to find educational toys or books for their toddlers, and we wanna do some research on that, we probably want to talk to.

Parents of toddlers or parents of toddlers who have recently shopped or tried to buy educational toys for them. So our screener survey is our first layer of, and our kickoff of research and figuring out who we want to talk to, who we want to take feedback from, and ultimately who we wanna make product decisions based off of.

So why do screener surveys matter? They have a huge impact, like I said, as the kickoff to your research. A huge downstream impact on the research you're doing, the feedback that you're getting, and the stuff that's factoring in to those product decisions that you're making, the needs that you're hearing, the behaviors, the pain points, all of that comes from research, but all of that research comes from people.

And ultimately, screener surveys are the main way or one of the most effective ways to pick those people and choose who you wanna hear from and what experiences you wanna learn about. Now, I love talking about screener surveys, weirdly enough, because they're one of those things in UX research that is, I think, often overlooked.

And something that, you know, a, the difference between a good screener survey, if you showed this to my mom, let's say, who's not well versed in UX research. She's gonna see a good screener survey and a bad screener survey. They're gonna look pretty similar. It's gonna be a few questions about the person, their experience, and their, let's say, background or context related to the topic.

But in reality, they could be very different and have a huge impact on the research. So the details matter big time with screener surveys, and that's what we're gonna cover here today. First by talking about the kind of downstream effect of it, and more importantly, looking at some of those little tweaks that can make a really huge difference.

Why screener surveys matter, A few missteps or mistakes can lead to a lot of, bad stuff for lack of a better word, going forward in your research. If you're not going through the details of your screener survey or making some of the mistakes that we're gonna talk about, that's gonna lead to recruiting the wrong participants for your interview, that might lead to getting unreliable or conflicting feedback that's gonna lead to synthesis sessions where you and your team are racking your brain.

Just trying to figure out why there's such a, maybe different opinion about something. And ultimately, it's gonna be leading to bad product decisions or uncertain product decisions, rather than being able to look at your research and say, Hey, we did the work. Here's what people want. Here's the need that they have.

Let's build something for them. And being able to do that confidently. The ripple effect or the downstream effect, as I mentioned, you talk to the wrong people, you get unclear, misleading feedback. You make product decisions on that. You waste time, money, and also lose stakeholder trust. You can be doing great research, you can have awesome interview questions, prepare, you can be doing everything right, but if the way that you're sourcing participants or the screener survey being a tool for that, if that's broken, that's gonna lead to everything else breaking down after, and a lot of headache and confusion when really our research should be leading to clarity, understanding and empathy towards our users.

Okay, so here's the scenario that we're gonna be walking through. It's called Run Buddy, a fictional running app. We're gonna take on the role of researchers at Run Buddy. Let's say it's an app that helps runners track progress and train effectively. So if any of you guys out there use, I don't know, Nike Run or MyFitnessPal or anything like that, think of it as like a, simplified version of that.

Our team as the researchers product team for Run Buddy, we're working on a new feature designed for people training for their first marathon. This is something my wife was doing about a year ago right now. And the inspiration for the workshop today and all the challenges that come up with deciding, yes, I'm gonna do it, I'm gonna commit to running a marathon.

What do I do next? How do I train effectively and how do I actually get to literally and figuratively the finish line? So in this scenario, we're planning interviews as the research team. We want to talk to people and we wanna learn more about that experience of training and preparing for the first marathon run.

So our interview goal is to learn about the psychological physical challenges faced by first time marathon runners or trainers during their training process and how these challenges impact their preparation and race day performance. And if you wanna take it a step further, maybe the goal is that we want to create a feature that is a specific training program in our app for people who have decided they wanna run a marathon for the first time.

And in order to figure out what that should look like and what's gonna be the most helpful, we need to talk to these people. So we're saying that we want to conduct interviews, but even taking a step back from that, we gotta start with our screener survey. We gotta find the right people to interview to get those insights from and to learn about those challenges and needs that come along with training for your first marathon.

Okay, so if you guys wanna head over to the fig or you could just look at my screen honestly. Totally. Nothing to do interactively here. We've got a screener survey example over here. We've also got one over here. This is our improved one. Don't peek at this one too much because this is gonna have some of the improvements that we're gonna talk through step by step, but I wanted to include it in the fig jam for you to check out.

But we have a screener survey right here, and this is basically gonna be our starting point for finding participants for our first time marathon interviews. And I will tell you that this is not the worst screener survey ever made, but it is not as good as it could be. And it would lead to a lot of those examples and those mistakes or missteps down the road that I just mentioned.

So the goal here is gonna be to walk through some of these common mistakes. Take a look at this screener survey example, the one that we're gonna use to find our interview participants and correct it, improve it, and identify some of these mistakes that can really throw you off when you're looking for people and looking to screen people for your interviews.

Okay, so we're gonna come back to that and start improving it. But let's look at the three big screener mistakes that we're gonna cover today. First one we're gonna look at no clear target. Second one, too many questions. Three vague or confusing questions. There are other mistakes outside of these ones that you can make in a screener survey that can definitely throw off your recruitment.

And I'll just share here, there are a bunch of articles down below in this fig jam. But this one here is an even further deep dive. I'll put this into the chat here and a couple of those other articles we'll dive even deeper. But these ones that I've highlighted here, no clear target, too many questions, vaguer confusing questions.

These are the big ones. These are the ones, the mistakes that I've made, plenty of times that I've seen when I've reviewed other people's research studies and screener surveys and the mistakes that I've seen. Get glossed over and lead to a lot of trouble or let's say confusion down the road. So we're gonna cover these, look at some examples and correct them together.

Let's start with the first one. No clear target. A screener survey is meant to screen in and out the people that you want to talk to, in our case the people that we wanna learn from about their experience related to. Their experience training for a marathon for the first time. So in order to create those screener questions, we have to have a clear idea of who should be in and who should be out.

That sounds and is obvious, but this is where a lot of screener surveys go wrong. You start writing questions you start think like maybe brainstorming all of these things that could qualify or disqualify someone from participating in the interview. Maybe if that's collaborative, you and your team have different ideas of who should be able to participate in the interview and whose feedback you wanna make those product decisions on.

There are a lot of factors that can go into it, but the simple fact is that if you are creating a screener survey to qualify or disqualify people from your research study you need to know, you need know, and be on the same page about. What qualities or what experience or characteristics you're looking for a persona could be a good starting.

It is a good starting point for this, and there are some other ways, but I'll get to an even simpler one in just a second. If you're unclear about who you actually want to talk to, you, screener isn't going to help you find them. Again, pretty obvious, but worth mentioning because a lot of times it can be, like I said, collaborative, different opinions, old personas, all these other factors that can come into play that make it more difficult than it seems or maybe more prone to error than it might seem to tie those screener questions to the actual people that you want to.

Talk to. So the biggest and best way to fix this is to create an Includes or an Excludes list. This is something that you can do with in addition to a lightweight or a even, robust persona. Or you can just do on your own. I called it an Includes excludes list. It could, you could call it a qualified disqualify list, but the idea here is to even strip down a persona even further and just get it down to a couple bullet points.

Who should be included in this study? What characteristics and which one should disqualify someone from being a part of this study? The key thing with your Includes or Excludes list. We're gonna see this as we go into some of the questions is that every question in our screener survey, everything we do after this, we should be able to link back to our Includes Excludes criteria.

It's gonna be our foundation for our screener survey, and it's gonna allow us to cut unnecessary questions. It's gonna allow us to ask better screener, study screener, survey questions that really target the things that we care about without any of the extra fluff or extra questions that will get into that can derail or bloat, let's say a screener survey.

Just coming over here to our example in Fig Jam, we have our scenario here where researchers at Run Buddy and we want to conduct some interviews to learn about the needs and goals and challenges of people preparing to run their first marathon. Here's my Includes and Excludes list. We wanna include participants who have trained for or completed their first marathon within the past two years.

And we want to talk to people who've primarily trained independently, maybe not as a part of a structured club or a, or with a professional like running coach or trainer. We wanna exclude participants who have had previous marathon experience. Obviously, if we're trying to learn about that first time marathon training experience, we wanna target people who have, recent memory or close proximity to that experience.

And we want to exclude people basing off of this point who've received professional coaching or trained as part of a more structured running club. Let me throw this out there to you guys in the chat and I'll give you, I'll give you a minute to answer. Is there anything that you would want to add to this?

Either in the include or the exclude participants. There's no right or wrong answers to this. But I want to just get this, get you guys thinking about it because this is how we would work as a research team, right? Maybe I would come up with a, one of us would come up with this as our starting point, put it out there to the group, and then have that brainstorm of what if someone did, X, Y, Z?

Would we want to talk to them or should we include or exclude someone who has this experience? So let me give everyone a minute here, check out what's in here and drop it in the chat. Is there anything, any other criteria we would want to either use to include or exclude someone from our study? And if you're not sure, drop it in the chat anyway. The whole point of this is to discuss it, talk through it together, and see what we come up with. So I'll leave this up for a minute.

All right. Chelsea said, exclude survey scammers. Robots. Definitely. Yes. Agreed.

And I'll say, while everyone else is thinking about this, Chelsea at Lyssna for participant recruitment, we want to w Panel quality is one of the most important things. Like when we talk as a team at Lyssna, the quality of people who are res or participating in an interview screening is super important.

That's like first line is getting no like spam, any accounts like that. Second is actually the quality of people, finding people who are really meet that target criteria. So absolutely right. We wanna make sure that whether it's like technical. Boundaries restrictions, screening or the survey itself that we want to have real people with real experience about training for a marathon for their first time.

Absolutely. Does it matter if they have previously used apps to help them in this way? That's, that is a great question to bring up, Gary. So I would say, let, let, we can decide on that together. I would say that my thought would be they should be included if they have, now here's, you might have a different opinion, but.

The reason I'm putting that out there is because we can change this, right? We can discuss it as a team. We might put this screener out there, get some interviews back. Let's say Kira, we wanted to block or disqualify anyone who's used an app or MyFitnessPal or something like that to train.

And then we put our screener out there and we're like, wow, we're not getting a lot of responses for this. Or, almost everyone we're disqualifying is, we're disqualifying them because they've used an app before. Hey, does it really matter if they've used an app before? If all we really care about is if they've had like professional guidance or a coach, does it really matter?

Maybe we could loosen up that screener survey a little bit. So to give a straight answer on it right now, my opinion would be let's include people who've done that as long as they haven't been part of, a. Structured or like professional running team or coach even if they've used apps and stuff in the past, that's good. We could learn from that experience. And I would say the pros probably outweigh the comms there. Prepare people that have used net. Yeah, I would even, I would agree with that. Kiri. I would say, yeah, we would want that. 'cause we would want to hear about their experience with it, if that's what we're trying to build. If we're trying to build an app or a digital program to help people train for their marathon, it's probably gonna be really helpful to hear about other things that they've used in the past. And for that reason, I would want them to qualify for it. But more important than the answer or the final decision that we would make now.

This is the exact type of discussion you would want to have when creating this criteria. And something you would wanna put out there with your team to get ideas like that, Carrie, to discuss them, weigh the pros and cons, and say, okay let's start it with this. Let's try it and let's see what comes back. And then we can decide if we want to tighten up those, that criteria or loosen it up as we go forward. Okay. So let's say maybe exclude people who run for fun without the goal to complete a marathon. Definitely, right? We wanna find people who, so this would, so I would say, Alexandra, I agree with you for sure.

I would say that would qualify in here, right? They've trained or completed their first marathon within the past two years. I would say that probably covers the criteria that you have there. But overall, yes, agree. We wanna find people. There's a lot of runners in the world and those who are training for their first marathon, and those who, who are just running for fun or to get in shape or to, for enjoyment, they are they're gonna have very different needs and behaviors.

So agree. Let's see. Michael said, are we targeting a specific segment? Maybe age 18 to 45. I'm coming right back to that one, Michael. Keep that in mind. That's gonna be in our next section here. Nice. Close family friends that will support them. So I think I'm gonna get to this in our third mistake in our third one, Becky. My question back to you would be, and take a minute to think about this, but what would we include or exclude? So if someone said, yes, I have close friends and family supporting me that would include them in the. In the inter in the study that would screen them in if they said, no, we didn't, and that would screen them out, what would that change about the why would that change their EL eligibility to be part of these interviews?

I don't know the answer to that. I'm actually, I'm just putting it out there to get your thoughts on it. It's not the first thing that jumps out to me, but I also am trying to think through it on the spot here. So main thing with all these questions, thinking about is this going to change their, is this going to affect the feedback that we're getting and considering for product features? I'll look for, I'll look for your answer or thoughts on that, and I think we're gonna cover something pretty close to that in in our third section here, Becky. Okay, so nice job with that and thank you guys for putting all your answers in the chat. Those are really good. That's the discussion I was hoping to have, right?

Seeing kind of a first draft of our includes excludes talking through it a little bit. We'd probably take longer if we were really the run buddy research team going through everything, but being able to discuss that, throw ideas out there and evaluate, what's really the bare minimum criteria that we want to include that is going to allow us to screen effectively for this study.

So what I have right here, while it might not be perfect, and I think there's some good ideas in the chat worth considering. It's a good start for us. And more importantly, what I want you guys to see is that it's something we can come back to and we will refer back to as our anchor for everything that we do going forward with our screeners.

Alright, so let's get to our next mistake. Too many questions. This is one of the, one of the biggest, and it's got a couple different flavors that I'll talk through and we'll look through with our fig gem. But one of the most common mistakes with the screener survey is to try to pack too much in there.

So with the screener survey, a screener is not a full survey. Surveys are a tool, a research method that we can use to collect data about our. Audience are participants. It's a way that we can we can use surveys in general as a way to gather insights about them. But a screener survey is not that, right?

A screener survey is just a filter. So the idea is we're not collecting data that we're going to use to inform our design decisions. Things that we're gonna use to, guide us towards certain needs or pain points. It's really just a filter and a screener, and we want to treat it like that. And a lot of times when I see screeners get overloaded, it's because there's.

They're packing too much into it and bleeding into that territory of research when really we wanna look at it strictly as a tool to get the right people in for the actual research, for the interview, for the usability study. So a good benchmark for a screener survey is three to five questions max.

On Lyssna, you're only allowed four screener questions for a study, which I think is great. I think you could have a lot of successful screener surveys that have. Two questions, one question and still be effective or actually be more effective to get the right people in? There are a lot of problems with having too many questions and I'll cover some of them, but one of the biggest ones is pretty simple, right?

Friction, fatigue, drop off. If you have 15 screener survey questions, people are go among other things, people are gonna drop off, they're not gonna finish it. It's gonna make your recruitment process more complicated and it's not gonna have that desired result of finding the right people to do the actual interview with. And the actual research with, okay, so here's some common culprits of too many questions. First one is. Demographics just, and maybe I would say just in case questions, if we come over here to our, and again, you can just look at my screen for this one. Our run buddy study example. Here we have age, marital status, household income, these top three ones right here.

These are questions that are you commonly see in any type of survey, right? They're there a lot. You might even think, like if I'm making a screener survey or any type of survey, we should lead with a little demographic background facts about the person where they live. I picked a couple here that are I think the most common. But not by no means all of them, these questions. Most of the time are not get included in a screener survey, but are not actually going to get you any closer to the right people. Now, there might be scenarios where age is a really important aspect of who you want to interview maybe marital status or household income. If you're working on a pro product for budgeting maybe household income or salary or something like that is an important screener survey. But if we were to look at these ones right here and then go back to our criteria down here or includes excludes, we can't tie any of these questions back. To our include or exclude criteria, they're just filler.

We're throwing them in when they really don't have an impact on who we want to interview. If our goal is to learn about someone training for a marathon for the first time, is it gonna matter what their household income is? It could, right? We could come up with a scenario where it could, but is this question really gonna be critical where we say, wow, if someone makes this much money, they're going to be, excluded from our survey. If they make anything under this amount, they're gonna be included. It's not gonna have much of an impact yet. A lot of times these questions get thrown in just by default. Even though they really can't tie back to your include, exclude criteria. Another common culprit, which I think let's see.

I. One of the questions in the chat was about this one. So Michael, that answers I hopefully your age specific one. But maybe Becky, I think this could relate to yours with close people, close friends that support them. Another way that screener surveys can get bloated is by asking the actual interview questions.In the survey, in the screener survey, again, we wanna really decouple our research from our screener. Yes, you can use surveys as a way to collect data about your users as a way to analyze or understand their behavior. Pain points more surveys can be used for that. But if we're talking specifically about screener surveys, this is a really common mistake that can lead to a lot of confusion.Downstream. If you're including interview style questions, things that you wanna, that you should be asking in the interview or the usability test, there's a high likelihood that you wanna take those out. The best way I can describe this is that if you're getting screener survey results back and you're already analyzing their answers beyond just can, should we include them in this study?

Then your screener has gone too far, right? So we wanna save these questions. These are deep dive questions. If we ask someone why did they decide to run a marathon? What were your biggest challenges when training? We don't want their quick open text answer that they're jotting down in a few seconds and using shorthand and bullet points.

We want to get follow up questions on that. We want to deep dive into their motivations, their challenges, what got them started, what it gave them, the idea that they wanted to run a marathon after, never running before or, seeing someone else do it. We wanna do that in the interview, right?

If you're seeing questions like this in your screener survey, there's a good chance that you want to take those out, save them for the actual interview, keep your screener survey nice and lean, and save all those deep dive questions for, a face-to-face or whatever zoom or listen to conversation about them with the actual person.

Amanda said here so you're saying we could ask the demographic info during the interview rather than the screener survey? Yeah. Or you could not ask it at all. You, the idea is if it doesn't relate to the criteria and your research goal, in our case, understanding the challenges of preparing for your first marathon, do you even need to ask about it? and I'm going a little bit farther and say, I don't wanna say you should never ask demographic info, but with these with these three questions I have here, Amanda, if we're trying to learn about. The physical, psychological challenges of training for your first marathon? Yes. Could this stuff be important?

Of course. Maybe if we're trying to then deep dive further and say for someone who's in their, 50 and over, then it's gonna matter, but these questions aren't really gonna or we don't know enough yet to say that these questions are going to matter or have an impact on our study. We're just throwing them in there.

He will get asked by stakeholders. Amanda you're that's how I know you're. You're really doing this every day. You will get asked by stakeholders when presenting findings. Sure. In that case, you could ask them about it on the interview. You could send a follow-up survey before the actual interview that's Hey, we just wanna know a little bit more about you.

You're in for the screener, right? So if you gotta do it, you gotta do it. And if that's something that's gonna be important to report back to, but I would also challenge you even, I know how hard this can be with, relaying it to stakeholders, but even offering a little bit of that pushback to them and saying, why does it matter?

If we're looking for if they're pressing you for, I don't know, whatever income or age you can push back with, why does that matter in relation to their challenges and behavior in training for their first marathon? Maybe you get a good answer from that. Maybe your stakeholders are like here's why we think it matters.

Okay, cool. Then let's include it. But. I hear you totally, Amanda. So you know if you have to do that, like I said, do it. Try not to let it bloat the screener survey, but also keep that in your back pocket, right? That question you can throw back to a stakeholder and say, sure, I can give you the ages of these people, but let's stay focused on what actually matters here, which is their challenges in preparing for their first marathon.

In which case, I would argue that a lot of these demographic questions don't really matter or they might matter, but they shouldn't affect whether we want to interview or hear from these people or not. There are other things that are more important that should disqualify or qualify someone from our research study.

The best way, as I've said to, eliminate these questions and this is a good strategy. Amanda, if you do wanna push back on that a little bit, is going back to that includes Excludes list. It's really simple. If you do a solid job on that includes Excludes list. And more importantly, if you're on the same page as your team or your stakeholders with it, here's who we're interviewing, here's who we're not interviewing.

Then you can point that to these questions and you can say hey look, this is who we're trying to interview. Does this matter for it? And when I said earlier with that includes Excludes list as being your foundation. This is the perfect example of it and the absolute best way to cut unnecessary questions from your screener.

We have some questions down here. Why did you decide to run a marathon? Does that have to do with this? Does that answer if someone's trained or completed their marathon within the past two years, or if they've trained independently? No. So if you have this and you're set on it and agree on it as a team or whatever, as a group of this is who we want to interview, you can always come back to this.

Whether it's to check yourself and to cut your screener down or to push back or, make that case. To cut things out of the screener survey, being able to refer back to an agreed upon persona includes excludes list is gonna be a really effective way to do it. Here's who we're trying to talk to.

Here's who we're not trying to talk to right now. Let's say, does this question affect any of these things? No. Probably a good case to cut that out of the screener survey. We don't need it right now. Maybe we can ask it in the interview, but not important for the screener surveys. Amanda, let me know if that was a helpful answer.

I sure hope so. All right, our last one here. So let's just go to this, right? So I would cut all of these questions if I were trying to improve this, and I'll show the final survey with this one. But I cut these and I cut these ones down here, all based on too many questions and too many things that are being asked that are not really relevant to the criteria that we're looking for. All right, let's go on to our last one here. Number three, vague or confusing questions and answers. If your questions are vague. People are going to guess, people are gonna give you vague answers. There's gonna be subjectivity involved, and that's gonna break your filter. You wanna have, if you've done a good job creating your include and exclude criteria, you wanna make sure that your questions and the answer choices that you're giving people actually reflect that.

So we have a question here, right? When did you train for your first marathon? Select one. I'm training for it now. I trained recently. I trained more than several years ago. I've never trained for a marathon. It's not the worst question ever. But as we'll see, there's some subjectivity in some wiggle room in this one that could be improved, that will help you source participants more effectively.

Some of the common culprits of vague or confusion, confusing questions. Number one is multiple questions bundled up as one when I've taught classes and workshops on this in the past. And I've given that, three to four, three to five question what's the word I'm looking for?

Checkpoint at or limit for your screener surveys, I've seen students and other researchers try to match multiple questions into one that is gonna lead to confusing questions, that's gonna lead to confusing answers and basically going to break the whole point of your screener survey, which is to get clear criteria of who should be included and excluded.

The other common culprit of vague and confusing questions are questions that have either, that are either subjective in what they're asking or are including subjective answer choices. So let's look back at this one right here. The fix being to ask clear, simple questions and offer specific straightforward answer options. Here's our first one. When did you train for your first marathon? I'm training for it now. I trained recently. I trained more than several years ago. I've never trained for a marathon. Not the worst question ever, but a, I'd say a realistically bad one or vague one. Mainly I've trained recently that can, I've trained more than several years ago.

This people might be able to a, give a clear answer to this one, but we're gonna have to do some interpreting on our end of their answers to figure out who is in and who is out from our research study. A better question and improvement on this one. Have you trained or ran your first marathon within the past two years? I'm currently training for my first marathon. Yes. I've trained for and run my first marathon within the past two years. No, I ran my first marathon more than two years ago. No, I've never trained for or run a marathon. We could maybe improve this one a little bit too, but there it's clear answer choices here. It's a clear question. It's time boxed. Instead of saying recently, we're saying within the past two years, that's leaving no room for interpretation here. So if we want relevant experience from our participants, if we want to hear people who have a fresh memory of training for and running their first marathon and we decide, hey, maybe that's within the past two years.

Let's just say that in the screener instead of saying recently or in the past or a while ago. Let's keep it super, super straightforward, both with the language we're using in our question, the options that we're giving for our choices and the time boxes or the specifics in this case, a timeframe of the experience that we're looking for. So with these vague questions, vague answers, number one, try to get it down to a simple question at a time. Don't try to mash too many questions together. Use clear, straightforward language and if there is any room for interpretation in it, like recently, see if you can improve that with something dead simple, super straightforward that people are going to understand, and more importantly that you and your team are gonna understand as a way to include them or exclude them from the study.

So you're not looking at these responses and thinking, oh, Jane said. She trained recently. Should we include her? Keep it really simple. If it's in the past two years, yes, you're in. If it's not, no you're not. If you've never trained for a marathon then we're not gonna spend time interviewing you to learn about your marathon training experience. Really simple. Really straightforward. And the common theme with this, and a lot of the stuff we've talked about, trimming down on the areas that might cause confusion, that might cause subjectivity or wiggle room and making it almost like clinically extremely straightforward and blunt in a lot of ways.

So your participants are clear on their questions and answers, and you're clear about who you wanna spend your time interviewing and taking feedback from beyond the screener survey. So this one here I'll just show you this example since we got about less than 10 minutes left. But you could do this one for homework. I left you some blank space here. How did you train for your marathon? I'd say this is an example of kind of a vague maybe the question isn't as vague but the answers by myself, with others, with some help, a coach helped me. And maybe you were gonna say this is gonna lead to problems, right?

Because if you say, with some help, and our criteria is we don't want to talk to people who have had a, been part of a structured running club or something like that. We're gonna get these answers back. We're gonna see people who have clicked with some help, and we're gonna have to decide on our own what does that mean? Should they be included? Should they be excluded? Again, a lot of wiggle room here. That, and a lot of ambiguity about the question and the answer. That's gonna lead to ambiguity about who we want to talk to in our interviews. So I'll show you how I improved this one. What resources or support did you use during your training?

Running fitness apps. Select all. Select all that apply here. Straightforward, simple question and answer. Running fitness apps. A professional running coach. Maybe this makes us, oops. Disqualify right here. My Fit Jam skill's. Not really working today. Let's see. No too much. Strike that out, right? That could be a disqualify structured running group or club that's a disqualify, friends or family trained on my own online resources. Everything else is gonna qualify, never forget. So we're asking a more straightforward question, giving more simple, straightforward answers, and more importantly, however we wanted to structure this, we have it, we're able to easily see these answers and without any debate discussion back and forth, we're able to say, yes, this person meets our criteria.

We wanna interview them. No, this person doesn't meet our criteria. We want to shift our fo or spend our time working interviewing people and taking feedback from people who do right, who have only used, let's say, a running app or, casual friends and family or help in preparing to train for their marathon.

All right, so since we're about of time here, I'm gonna go through a quick bonus section and then we will do a super short wrap up with some key takeaways, a little checklist, and get you guys outta here to enjoy the rest of your day. Good question, Elaine. I'll get to that in a second. Another thing that I wanna highlight is, we focused a lot today on writing good screener questions, making sure that we have a clear idea of our target audience, making sure that we don't have too many questions and we're trimming the ones that don't really relate to our criteria, and making sure that our questions are straightforward, clear, and easy for us to parse of who we want in and out of our survey.

But some other things that you want to include in a good screener survey are some of the details surrounding the question. Basically some ways to create a good. Participant experience outside of just the questions that you're asking them. So some other good things to include in a screener survey.

The goals of the study, the time commitment, the compensation that you're offering, privacy logistics. And if you look at the improved one in your template up here, you'll see I have all of this. And you can use this as a template for it. It's giving the goal of the study. It's saying that it will be, that one got cut off a little bit shared with the research team. It's giving some info and compensation. These are all things that don't really have to do with who's screening in and who's screening out. But it's gonna set you in. In the spirit of this whole workshop and this whole recruitment process, it's gonna set you up for a smoother research process. Not only are you now recruiting the right people with improved questions, but you're also setting the expectations for the study so that when people get on you with you and wanna do that interview, they're gonna know what to expect and what to, and what they should be sharing, and maybe even have some ideas beforehand.

It's gonna reduce things like tech problems. It's gonna reduce things like, oh, I didn't know that's what we were talking about here. A lot of other things that can create hiccups in the recruitment or research process are gonna be eliminated if you can set the right expectations in your screener survey. Give a little context and basically introduce people into the purpose of this screener survey and what they're opting into, if they wanna be a part of it. Okay, let me share this example. These are some good questions in the chat, so I wanna get to these. The last thing in terms of creating a good screener survey experience, or a good recruitment experience I'll share a personal example.

This is a screenshot from an app called FreshBooks. Basically, FreshBooks is an a platform for freelancers to send invoices, stuff like that. I've used it for a really long time. I got a. Email probably like a year ago that had a screener survey from FreshBooks and it said, Hey, we're looking to interview people about how you use FreshBooks to invoice, clients, stuff like that.

And I thought, what the heck? Let me fill it out and I will I'll be part of this research study. I can tell you just based on the screener survey questions, they were really good questions. And I could also pretty confidently say, although I don't know this for sure, that I would be a really good participant, I would've screened in to, or I did screen into this study and I would've been a really good participant based on the criteria it looked like they were looking for. I probably would've been a good person to interview. So I opted into it, and when it came, I got an email that scheduled the interview with someone, a researcher from FreshBooks and g when it came time to do the interview. No one showed up. I had to look for the link of where I was supposed to join. I didn't know exactly. There were some time zone issues. I was basically volunteering my time. I think they were offering compensation, so not totally volunteering it, but I was saying, okay, I'm in. I'm screening in. I'm the right person to talk to. I'm gonna give valuable feedback on this or the best feedback that I can.

And when I showed up for the interview no one was there. And in addition to that, leading up to it, I wasn't really sure about. I. The scheduling, the time, the process, I just blindly showed up and hoped someone was there, which they weren't. Now this isn't to rip fresh books or anything like that, but what I want to also emphasize in this idea of a smooth participant recruitment experience is that again, it's not just all about the questions if you have a great screener, but no follow up.

If there's a missed interview, if people don't know what to expect going into it, that is gonna lead to some frustration, a bad brand moment, which it was for FreshBooks, even though I still really like it a lot. And it's gonna lead to more issues with your research. No shows confusing interview responses, stuff that's gonna lead you astray from the actual purpose of understanding, in this case how people, invoice clients and, use FreshBooks to manage their finances.

So one thing I wanna highlight is in, in addition to screener surveys, Lyssna has these features built in emails that follow up with people once they screen into your study, that give clear direction on where to be, when to be an introduction to what the topic of the interview is. All of these kind of admin tasks that honestly suck to do manually. But that make a really big difference in, in your participants showing up, feeling comfortable, confident, ready to give you good feedback versus feeling. Shaky, unsure, or potentially like in this case, getting ghosted when they volunteered their time to do it. Topic of today's workshop was all about the questions and the screener survey, but I wanted to talk about this just at the end and include it in that fig jam so you can see and, take some time to digest after this workshop about some of the other aspects of a good recruitment experience outside of just the screener survey questions.

Alright, so let me get to some lemme get to some questions here in the chat. If you guys don't mind sticking around for five, five more minutes. I'll answer some of these questions. And then wrap up and send you guys on your way. Let's say Amanda. Okay. Elaine said thoughts about the order of questions in the survey? That's a good question. I think. I am gonna give an obvious answer here. I don't think they ma it matters that much unless you're building off of the question before it. There are also instances where you could use like branching logic if you wanted to say, if they answer yes to this question, answer show them this question that obviously would affect the order of it.

I don't really have any great tips outside of just the obvious ones that if a question is based on the one before to that would affect the order that much. And if you're do, I'd say my main point is if you're doing a good job of cutting it down to a few simple questions, in most cases the order isn't going to matter as much.

And if it does, you'll probably spot that pretty quickly. I'd say my only other piece of advice there would be. Like start with if the order truly doesn't matter in terms of questions leading to other questions, start with simple, easy to answer questions, a quick multiple choice, one maybe followed by a, open text question towards the end. But I would say in most cases, if you do a good job of cutting down the amount of questions in the study, the order is not gonna matter that much. And I would have to s maybe I could find an example Elaine of one that would be really that would be, a good example of this. But in most cases, if you can do that there, I don't have any super relevant tips or tricks that are gonna drastically affect it.

Depending on the order. Again, outside of just an obvious, if one question's based on the one before it, you want them to go in that order. So it doesn't matter that much. Lou, what's the ideal number for screening questions? As few as possible. And I would say five max. Like I said, listen to limits you to four. You should be able to screen a, you should be able to get it down to that. And I think three is really the sweet spot. Do you vary your approach if it's design, test, or surveys rather than interviews? Any considerations? Good question, Carrie. I think the types of questions you would ask maybe for a short answer is no, right? The approach is everything we covered here is going to be the same, whether it's or these. Rules of thumb or best practices are gonna be the same, whether it, regardless of the study that they're opting in for it's really just gonna be the content of the questions. That might change maybe for a lighter study.

You might have I shouldn't even say, I was gonna say maybe for a lighter study you might have some lighter criteria. But that's really just saying it's case by case. But short answer is no. You wanna keep it short. You what? Regardless of what study you're running, you wanna get the right people for it.

And things that I showed you here are gonna help you get the right people for it, and that's not gonna change. Regardless of what the study is about or the method that you're using, you wanna keep it short. You don't wanna overload it with too many questions. You don't wanna put interview questions or usability test questions in the screener, and you don't wanna have vague or confusing questions.

And that's gonna be an across the board, yes. You can reuse screeners in lesser. Yep. Cool. All right everybody. So to wrap up here's a little checklist for everyone. When you're doing your next screener survey, do you know who you want to talk to? If you have a lightweight persona, you can use that to create your include excludes list. That's your first step. That's your anchor for everything that you're doing going forward. Do that first. Make sure you and your team know who you want to talk to. And if you have, that includes Excludes list, you can refer back to it over and over again to make sure that your questions are on point and you can adjust it if you feel like, hey, we need to maybe loosen up on this criteria or tighten it up as you get some responses in.

Do all the questions help qualify people in or out. Again, using your Includes excludes list or qualified disqualify list as a foundation. Go back to that. Look at it. If you have 10 screener questions, look at that criteria and see if there's anything that can be caught. Are there any questions in there that don't re directly relate to it? That's why I drew an arrow from this to this, right? Literally do it this way, right? You're like, okay, what? This is criteria, what ques maybe do it reverse, right? Here's a question. What piece of criteria does this tie back to? If it's none of these, that's probably a good sign that you can cut that question.

Are the questions short, specific and easy to understand? And are you saving the rich answers, the rich questions for the actual session, especially if you're doing an interview. I know we packed a lot into this hour and seven minutes here, but I wanted to leave you with this checklist at the end.

Covers everything. Refer back to these slides. But if you can do these things that I'm showing right here and run through that checklist on your own, starting with that foundation of your persona or your includes, excludes criteria, that is going to make a massive difference in the quality of your screener, surveys, and everything that comes out of it, it's gonna save you a lot of wasted time in potential research and synthesis from talking to the wrong people.

All right, everyone, thanks for sticking around. Have a great rest of your day. Thanks so much for joining, and keep an eye out for some upcoming workshops soon. All right, see you later. Everybody. Have a good one.

Workshop overview

About the host

Try for free today