Hello. What's up everybody? Welcome. So thank you for being here, everybody. Let's get into it. My name is Joe. I'm going to be. Your guide to improving your research questions today. I'm a UX researcher, product designer, and my role here at Lyssna is Design Advocate, which is basically through things like workshops, working with our clients, customers, a whole different bunch of channels.

My main goal is to help you guys be successful in your research initiatives and doing that with Lyssna. So through that I get to look at a lot of different research that's going on. We do a lot of research. I've done lots of research and run into a lot of the same problems that we're going to talk about here today.

So I'm going to try to bring that experience and these real practical examples into our lesson here today. And as I said, give you some clear, specific tools to run better studies and more importantly, get the impact out of them. Be able to turn that, those really great insights into actionable improvements, not just these bits of information that never really go anywhere, which happens unfortunately a lot with research.

 The title of this workshop I tweaked this one a lot to try to get the point across of what we're doing here but not make it too wordy, which I think it is a little bit, but how to design research studies that produce clear actionable results.

Here's what I kept coming back to, and that we're going to talk about a lot today. Here's what I wanted to name this workshop. How wishy-washy research leads to wishy-washy product decisions, vague, unclear, passive research questions leading to results that aren't really actionable and wishy-washy was just the phrase that I kept coming back to in this.

So I didn't know if that would work for a title. Maybe at the end of this workshop, you guys can let me know in the chat if, if we run this again. Should we just call it this? Either way, whatever the title of the workshop is, the point of it is to identify what wishy-washy research is, the types of questions that lead to kind of these vague results and ultimately result in all of that work, all of that research, all of those insights, just going nowhere, right? Maybe getting debated in meetings, maybe having circular kind of conversations and interpretations about it. But yeah, wishy-washy research, wishy-washy product decisions. We're going to learn how to identify those questions and how to improve them and turn them into questions and research studies that are going to lead to clear action, alignment, improvements to your product.

Okay. Research should keyword lead to confident product decisions. If you're trying to decide should we build feature A or feature B, what are we going to spend our next, two months working on research should. Get you there and lead you to an answer that your team is aligned on that you can point to.

Real user feedback to justify research should lead to actionable improvements. As I've already said several times, research isn't just a, intellectual pursuit or curious pursuit to learn more and deepen your knowledge of users. It's an ultimate goal of being able to use that to improve their experience, to improve your product, to make updates to your product that are going to give users the capabilities they need that is going to improve how successful they are towards their goals.

And finally, research will lead to a shared understanding a baseline level through your team and. Maybe even other areas of your organization about your users, their needs the feedback that you're getting, the specific criteria that you're basing decisions on. I know some of you're probably looking at this yes, we know Joe, this is how it should work.

But I think it's important to set it up that way if we're going to talk about what it often leads to and how to improve that, which is more debating, more interpretation, more indecision and inaction. And I see this all the time in projects that I've worked on teams that I've worked with all who are.

Competent and, good teams of people, talented people working towards the right goals. But you see this so often where the research just doesn't go anywhere, right? You get those results and it leads to a lot of long meetings with people trying to figure out what to do next.

Has anyone has anyone experienced this? Give me the thumbs up in the chat. Maybe not thumbs up, but let's see. Yes. Research. Yes, Audrey, we're going to talk about that. So we know how it should be, right? We want confident product decisions and we know the reality, which is that it leads to more kind of circular conversations.

Now, I'll say research can certainly lead to discussions, questions and a level of, figuring out what to do next. It's not like every research study you run is going to lead you to a clear, definitive rock, solid answer of what to build next. There is that level of interpretation, discussion, further research, right?

But what we're talking about here is really the core that research is often leading to more speculation than actual competent product decisions. So there are some ways that that we know of that can lead to what we just described, right? Research results going nowhere. These are the usual suspects, right?

Biased questions, bad sampling or screening right? Not, or recruitment, not finding the right people for your study, using the wrong methodology for a certain study. There are all of these, flaws that can happen in a research study that can lead to these further speculation and inaction from it.

We're not going to talk about those as much today. These are the fundamentals of a sound research study and probably things that most of you here are already familiar with and can already point out or are not including making sure to, screen out of your research studies. But today we're going to focus on something more subtle our wishy-washy questions.

These are questions in a research study that, let's say they're not outwardly wrong, right? They're not on the face, just biased leading, poorly worded. They look like valid questions to include in a research study. And I have certainly looked at research studies, created research studies that had questions like this in them, and they've looked good to me on first glance.

And I've said, okay, this is, this study's good to go. With more experience and seeing the results and like what happens after that. I have got a sharper eye, let's say for some of these questions and how they can slip through the cracks. So we're going to talk about what these are and talk about how to spot them in your studies and how to eliminate them.

They're not biased, they're not wrong, but they're not useful. They don't lead to clear confident product decisions. So we're going to go through some of those questions or types of questions and improve them. But why do we see these types of questions? Some of you in the chat have already pinpointed a lot of these.

Audrey, you got one here. Oftentimes the research is used to prove the team's hypothesis, right? We did the work and we don't want to change it. Absolutely. That's going to be my last one. But we see this a lot because. Some of the common reasons, too many cooks In the kitchen scenario, you create a research study.

It goes through stakeholders, teams, people with other goals or motivations for doing that research. It gets reviewed, it gets edited. This is not unique to research studies, right? This is everything. When it passes through too many rounds of reviews without that shared understanding and everyone's got a hand on it things tend to get a little bit more, kinda a Frankenstein type research study or product or design, whatever you want to insert in there.

This is definitely a big root cause of these wishy-washy research studies and the wishy-washy results that come from them. The other big reason is no decision criteria upfront. The phrase that I think about with this is, let's see what happens, right? Now, there's I'll say there's some scenarios where that might be a valid way to approach a research study, but if you're really trying to use research to drive a good product decision or a direction, let's say, something that you're going to invest time in and you don't have that decision criteria upfront, that is going to be a problem later when you have to figure it out after the fact.

What I'm talking about here is being, having a shared understanding before launching a study about how you plan to interpret these results. What are the goals of this? What level of, if we're running a study to evaluate feature A versus feature B, what criteria is going to allow us to make that decision?

What sample size? What if we're quantifying it somehow, what is that threshold that allows us to say, yes, A is the winner. That's what we're going to spend our next couple months building, or no, we need to do a little bit more research to get there. We're confident enough to do it. If you align on this beforehand, then.

Theoretically, there's no debate once the results come in. But oftentimes that decision criteria is wishy-washy in itself, right? Where it's let's see what comes back. And then that leads to everyone running wild, trying to interpret those results and figuring out what it means with no real standard set of how these decisions are going to be made.

And this is the one that everyone is hitting on in the chat or a couple of you're hitting on in the chat. And it's true, just avoiding uncomfortable answers. And, research the company line for research is we want to know what our users think. We want to make user-centric decisions, we want our assumptions to be.

Invalidated and get the info we need to move in the right direction. That's talk in the talk, walk in the walk is a lot harder, right? When you actually really commit to putting something out there for feedback that you have invested time, money on, and maybe already decided that's the direction that you're going.

A lot of times that will influence how the research is done to avoid something that could possibly. Blow up those assumptions. Sounds counterintuitive. I've seen it over and over again and I've probably been guilty of it at some points where I've let that, let some previous work that we've done or some kind of predefined direction influence that research in one way or another.

So these are not problems with, bad researchers or people who don't know what they're doing. It's a bunch of factors that lead to these kind of watered down research studies that don't really go anywhere. So what we get wishy-washy questions, wishy-washy results, wishy-washy decisions. Let's look at three common types of wishy-washy questions.

These are the ones that I see all the time. I tried to, like group them by some common characteristics or themes. But between these three types, which we're going to go through in detail, these are the most common and they have a huge influence on how actionable those results are. So the idea here is to go through these, define them for you, give you an idea of what to look out for and why these types of questions are ultimately going to lead to inaction and more confusion and also give you guys the tools to fix these and improve them.

Coming out of this workshop, hopefully, if you've got a research study running on Lyssna and you've got some questions written up, you'll be able to go back through those, see if you've got anything wishy-washy in there, and be able to correct it.

Or the next time you have a stakeholder meeting and everyone starts chiming in and you can see it getting watered down a little bit, you can. I don't know, pull up these slides or a recording or cite some of these examples here and steer everyone in the right direction towards a study that is going to get you the answers that you're looking for or that you need, maybe not what you're looking for, but what you need.

All right, so we got our three common types, misleading metrics, make believe questions and safe subjectivity. Let me check the chat to see how we're doing here. Tatiana, let's do a quick test, if I hear it one more time. Totally Tatiana there, and I'm a huge fan of, let's do a quick test, right? But you have to set what that, what's the point of that quick test?

What are we deciding on based on this quick test? How are we measuring the results? How are we presenting this question or this thing that we want to get feedback on for that quick test? I'm all about quick testing, putting stuff out there, testing early and often. And I think that's what you're getting at though.

The key there is that it still has to be thoughtful and intentional, right? If you're just quick testing everything, and that's your pass to just throw a study out there and see what happens. As we know, again, that's going to lead to a lot of zoom meetings that take way too long discussing research results.

All right, so let's get into these three common types. We're going to start with misleading metrics first one, okay? So misleading metric type question, let's say is a question that is rooted in this idea that quantifying something is going to lead to clarity. I think for a lot of people. I don't include myself in this, but for a lot of people there's a level of like safety and certainty in numbers.

I shouldn't say, I think that's the case applies to me too, but maybe I'm a little more flexible with research because I've seen it fail in some cases. But there's this level of comfort in seeing, for a question like this, right? How well do these recommendations match your criteria?

If you're looking at the results in that, there is a level of certainty. If you see a 3.5 and that's the average. Or 80% of people gave this a four out of five. It's clear, it's direct. There's, you're not digging through interview transcripts. You're not, trying to dig deeper into the user's motivations.

You're not comparing it to background questions, you're not putting context to it. It's straightforward. And I actually really agree that there's a level of appeal to that, to just see, what is the concrete number to this? And having that clear criteria to measure and make decisions on. And of course there is lots of space for quantitative questions or quantifying things like this question here in UX research and many areas where that can be extremely helpful and can improve that clarity and that directness that you're using to make decisions.

But when it comes to this category of misleading metrics, there are often a lot of examples where a. questions are trying to quantify something and doing it. But that quantification is not really leading to more clarity, more valuable information to make decisions based on. Bryce, to stakeholders

Joe, from Bryce, I would be honored if you want to pull up any screenshots from my workshops to bring into a stakeholder meeting. I would be more than honored. Let me know how it goes. All right, so let's look at an example for misleading metrics.

Let's say we have this product, an app that is an event space marketing. Marketplace. So something like Pure space, Liquid space. I've been using a couple of these recently, so it was top of my mind. And let's imagine that there's an onboarding experience or some, bunch of questions asking you about the type of event that you're hosting with the goal of recommending you your top event spaces.

Given you just a overview scenario here to help set the scene, but maybe we're designing this feature of asking about people's events and recommending like the top event spaces. Maybe we want to do this to improve the conversion rate of booking them. Maybe we're seeing people are spending a long time looking for an event space and not booking one.

So we can imagine the business goals related to this. But the idea is we've created this event questionnaire onboarding. We're recommending, let's say three spaces based on your preferences and your event. And the goal is to provide really good recommendations there that people are going to book. So imagine we're testing this out or running some sort of study on it, and we're trying to evaluate how strong these recommendations are.

Are we successfully delivering really good recommendations, ones that people are going to end up booking based on the criteria we're asking. Or are they feeling like it's just like popular spaces or things that maybe look good but aren't really relevant to them. So here's an example of a misleading metric question.

How well do these recommendations match your criteria? So let's say someone walks through the onboarding, they give, put some information about their event. They see these recommended spaces, and we're trying to evaluate, how good those recommendations are. We could ask a question like this, but when we go to review the results, right?

If we see a 3.5 out of five was the average, how confident do we feel about saying this is, we got this, we nailed it. Probably, you're probably feeling okay. It wasn't terrible. It's right there in the middle. But just the way I'm talking about this right now gives you all the information you need to know.

If I'm thinking about it like this, imagine a meeting with, five people on your product, team, stakeholders, everyone trying to look at that number and say, should we spend the next couple months launching this onboarding or this recommended feature? Is this the thing that's going to help people find the perfect space to book?

I would even argue that even if you see a, the average is a 4.2 per, something a little bit higher are you feeling 100% confident. Are you feeling like you truly understand the or really have a good read on how effective these recommendations are? Maybe the better question than all of these is there going to be a debate about the results, whatever they come back with, and are you going to have clear next steps to either improve it or make a decision?

I think the answer there is you are going to be debating this. Whatever the average comes back on this one, you and your team are going to be, everyone's going to have a different opinion about how good that is, what that means, if it's good to launch, what you should improve, and it leaves you in that limbo and those circular discussions of figuring out what to do next.

And I would say most likely, in a lot of cases I see it's not going to go anywhere. So let's look at a better example. Here's the question I wrote up. Considering your event space needs, how confident are you that these recommendations are a good fit? Very confident, I would book one without any more browsing.

Somewhat confident, I'd consider one of these before browsing further. Not very confident, I would likely keep browsing for other options. Not confident at all, I wouldn't consider booking any of these and a space for other. Now this might not be the perfect question and there are almost unlimited other ways that you could improve on this one here.

But the point I want to illustrate with this is that the level of specificity you're going to be able to look at these and not have to interpret what a 3.5 out of five means. You might see a spread that really indicates specific intent or specific evaluation of those recommendations. These are specific answers they give, possible and realistic next steps.

They're specific in terms of what a user would do or how they would act on this, which is ultimately what you're trying to figure out, right? So by doing this, yes, you get a clear number of, a one through five and you can gauge, relatively well if this is, a slam dunk and really great, or if it's totally missing the mark.

But with a question like this, still simple, still easy for the participant to answer, but much more specific in the much, much less room for interpretation, let's say in terms of the answer that they're providing. Whatever it is out of these ones, this is also going to set you up for a much more much better next steps, right?

So if you're debating what to, what research do we want to run next on this? What should we try testing out? If you want to ask follow up questions to this one, you're going to be much better set to do that when you're given specific answer choices like this one, rather than trying to figure out what that, four out of five really means and what to do with it.

So for misleading metric type questions, right? You want to tie it to specific behavior. Look for these types of questions in your study once that rely on quantifying something and ask yourself, yes, this quantifies something, but does it really give me a clear understanding of how users feel about this?

Does it set me up to ask follow up questions to make a decision to evaluate how well this is working? If the answer is yes, keep it. But if not, consider rewording it or changing that question type to something that is going to pinpoint their actual feedback and reaction and emotions towards it, rather than just relying on, that cold, hard number.

So when we see a misleading metric question, we want to. and we're changing it. We want to tie it to real behavior. We want to be specific. We want to have clear segments or options offered, and ultimately ask the question, is this going to give us information that helps us make a decision or get closer to a good decision?

All right let's, I'm going to hold on this example here and we'll go on to our second one. Let me pause for a second. Before we get into our make believe questions, I'll give you guys a minute in the chat. If anyone's got any questions on our misleading metrics, drop 'em in there. I'll take a look at the chat.

Let me know if you've seen questions like this. If you've written questions like this. Give everyone a minute to chime in the chat and then we'll move on to our second type. 

I am just looking at your chat, Sammy. Since I'm new to research, I definitely have a lot of hindsight when looking at my results. I should have worded this a different way that, you'll, I don't know how new you are to research, but I'm not new to research and I do that all the time. And that is a good sign, right?

Research is, no matter how experienced you are, the whole idea of it is to learn things that you don't know, right? If you think that you can put a research study out there, and it's going to be, every question perfectly worded, pinpointing the right, things that you want to learn about and like predicting everything that can happen.

It goes against the whole point of doing it. But the key there is being able to go back and look at it. And that is super relevant to what we're talking about here, because a lot of times that doesn't happen, right? It's okay to put a wishy-washy study out there, but when you get the wishy-washy results back, the important thing is to say, these are wishy-washy.

We got to fix the, we got to go back and look at Joe's slides and fix these questions, rather than saying let's figure out our next big feature based on this. Or let's have a two hour zoom meeting and FigJam session to, posit our ideas and opinions about what this means. So Sammy that's a sign that you're a good researcher, a thoughtful researcher, and doing the most important thing, which is improving how you do it.

Are there any cases where a one through five scale question wouldn't be wishy-washy? Yeah, absolutely. The types of. Yes. The short answer is yes. Let me think of example for that. So I'll, I'm having trouble coming up with a super specific one off the top of my head, but here's where they're not wishy-washy.

One is, if you as a team have an informed and defined criteria of what that, one through one through five means and how you're going to evaluate it, right? It can also apply to things like, an NPS study, which is one through one through 10, but there are, that's a good example because there are clear, there's a clear framework for how that question is posed, which is, would you recommend this to a friend? And how you can interpret those results. A lot of times, I'll say also where these types of questions can work really well is in conjunction with other questions like the one that I just showed.

So where you're getting both, you're getting some more raw or specific open-ended feedback. And then maybe like tying that to, in addition to that, tying that to a more like. Straightforward one through five ranking. It can also be with like an agree and disagree question. So it's yes, there are tons of use cases for them where they're not wishy-washy.

I'd say my main answer to that is if it's the only thing that you are relying on and you don't have any more specific context about like how people are rating that then that's where you get into that territory where you might be, putting too much confidence in just a number without really understanding some of the background or like the actual question you're trying to answer.

We've got more questions here, Kesh kinda issues. Always have someone else on the team review your study before it moves on. Yes. Audrey said example for Lee's question, yeah, let me see. You got an example. If the goal is making the user feel safe as in solid of a security app, how safe do you feel?

Yes. And you can follow that up with another question that's what informs your answer? What if you rated it a one, what made you feel unsafe? What would help you feel more safe? So yeah, there are definitely I use questions like that all the time in my studies, but they're usually surrounded by other questions that are going to contextualize that number, right?

Misleading metric question. What is the essential first state take to make sure the new question actually correlates with the user? Great question, Paolo. We'll wrap that section on your question. The essential first step is to write the question, put a couple maybe do a little pilot internal test with a couple of your team members, something that doesn't really matter, right?

But just get some results. Then look at those and say, are we going, is this what we're going to be able to make this decision on? Forget about what the actual results are, but just looking at those and visualizing, a couple days down the road when you have actual results for it, if you were to launch it are, is this going to be effective?

Are we going to feel confident? What if we get a three out of five across the board for this? What are we going to do next? So that is the essential first step is seeing it. Bringing it up to your team or addressing it and clarifying the success criteria, how you're going to be evaluating this and what those next steps are going to look like.

And one of the best ways to do that is to just, especially in Lyssna, show the results and say, this is what it's going to look like when people, when we have a hundred people complete this study. Is this what we're going to decide our next feature on? If the answer is yes, and you come up with that criteria, awesome.

If the answer is good point. I don't know I don't feel very confident in that. That's a cue to reword it or rethink how you're measuring or evaluating it. Let me know if that's helpful, Paolo. Great question.

All right, let's get into our next one. Make believe questions. Okay. So make believe questions are another type of wishy-washy question that ask users to project into hypothetical scenarios and over extrapolate what they're a situation that they're in or how they're responding to a question.

So I'm saying make believe like in almost like role playing. You're putting your participant into a scenario that is really specific and maybe, as you get more specific with it, not as rooted in their day-to-day experience. A question like this one imagine you're hosting a hundred person corporate event with catering, AV setup, breakout sessions, and a tight schedule based on this page, what, if any, additional details would you need to decide?

Whether or not to book this space. Now this is a really like wordy long question. This has some other issues to it, but the idea here is if we're asking people about people who plan events and book event spaces to give us feedback on something, this question is giving them a lot of criteria to imagine.

It's giving them a lot of roles to play or things to consider that they might be more focused on when answering the question, of answering it from that perspective rather than their own experience. So make believe questions are where your question is loaded with lots of setup. Imagine that you're doing this, lots of criteria.

Put Forcing your user to envision themself in a different scenario. That and a specific scenario that gets them away from their actual experience. So it adds complexity that doesn't really reflect real decisions. Their responses become unreliable. Insights drift away from real world behavior and they're hard to act on with confidence.

You don't know how well that scenario or how aligned that scenario is with their day-to-day experience, and you don't know how much they're answering from their own experience or how much they're answering to play along or or to go along with the criteria that you're providing them.

So let's look at another example. Event place event space, marketplace. We could think of the same one that we were just looking at, but this time we're looking at an event space, detail page. And maybe the decision, or what we're trying to learn from this research is do these pages include the right information for someone to confidently choose a space?

If someone's browsing for a spot to host their next event, are these pages giving them the information they need to do that? And if not, what's missing? Here's our question that we just looked at. Imagine you're hosting a 100 person corporate event, catering, AV set-up, breakout sessions, and a tight schedule.

Based on this page, what, if any, additional details would you need to decide whether to book this space Now, someone who is, let's say an event planner who's frequently running events like this this is going to be an okay question for them, right? It's close enough to their real experience that they're going to be able to answer it and say, I always look for something, like this.

Or I look at the number of outlets and where they are, or if there's a coat rack and stuff. So they'll be pulling from their own experience, but someone who maybe is an event planner but runs a different type of event maybe does not run something with, an AV setup or something like that

They're going to be able to answer this and probably put themselves in that role to see what's missing. But they're going to be very far off from giving you the feedback of how they would evaluate it, if they were hosting a networking event next week and looking for a space. And a lot of times when I've seen questions like this and done them as a participant and walked through them, you can really feel yourself trying to complete the question and the role rather than offering your own experience.

I can think of a couple recent studies where I was kept looking back at the question and thinking like, am I, am I like hitting the right criteria? What is the scenario again? What should I be looking for? And it's more like figuring out a puzzle than actually saying how I would choose an event space.

So one of the best ways to get out of make believe questions is to ask some questions beforehand that are going to put participants give you some context about participants and their day-to-day experience or their reality and ask more straightforward questions then based on that, right? And so instead of the giving them a role and criteria, ask them about the things that they do.

For this one, maybe a question like, in the past year, what types of events have you booked event spaces for? And we can get an idea of the types of events they've done, thinking about how you choose event spaces, rank the following factors from most to least important. Again, there could be a million different like types of questions.

These are just a couple examples. But the key here is that if we ask these two questions before we get a good understanding of this participant in terms of the type of events they host, the things that they care about, and what they're actually prioritizing in their day-to-day life, or let's say role as an event planner.

So if we ask those questions, then when we get to this one, thinking about a recent event you planned, what if any, additional information would you need to decide whether or not to, shouldn't say booking the space, whether to book this space? So then when we get to that question that was previously this complicated imaginary one we're asking them about their preferences, the things that they care about, a recent event that they've hosted and making it much easier for them to pull from their own experience.

The key here is we still have information and context about their the types of events they plan, the things that they care about. So we get to ask them then, based on their experience, get their real feedback, understand some context behind it, and pretty much not load everything into this one complicated story or role that we're asking them to take on.

Okay, let me check out some questions here, 'cause I see the chats popping off. Okay. So how do you make avoid making a tree test or a navigation test prompt to make believe. That is a tough tough place to that's a tough one and I'm, that's something I'm working on right now too, so I was editing questions with that exact thing in mind recently, Sammy.

The idea is first you want to make sure that you are recruiting people who doing a good job of recruiting your target audience. So for this event planning example, right? If we're doing a good job of, let's say, recruiting people who regularly book event spaces in their professional role who are meetup organizers or something like that already, right there, we're going to have less of a need to play make believe or make a make believe question, right?

We're already we're not asking them to imagine that they're an event planner or planning an event. They're already doing that. So I would say that is the biggest one. And it is challenging with tree test and navigation. But the less that you have to put people in a role the better.

So for navigation and tree test, maybe when it comes to recruitment, that is something like, I don't know, let's say booking a hotel, right? Let's say that's the task that you're doing and you're asking someone to do tasks related to that. In a tree test, you want to find people who have recently, booked a hotel for a, multiple hotels over a short term trip.

You want to find people if they're booking it for people in their company, like corporate travel people who have done that recently and are familiar with that task. So the, I'd say the biggest thing there is recruitment, right? Finding people who really. Have, don't have to play, make believe that much who already have that real experience where you can then give them a scenario that is going to be at least somewhat related and familiar to them that they can pull from their own experience on.

All right, let's check yes, there will be a recording. I'm going to do this guys. Let me just share 'cause I know we got, I'm going to run through our last one pretty quick and we'll wrap up. But I wanted to just share this promo code here. I'll put this up at the end, but if anyone has to run check this out, that gives you 20 free testing credits on Lyssna go and put that in right now and you can, get those in your account to recruit with. I'll share this in 10 minutes. Again, put it up on the screen, but if you got to bounce a little early claim that code. So you got some credits to put this stuff to work.

All right, let's go to our last one. Safe Subjectivity. This is one that a lot of you guys hit on just in the intro.

Using vague passive or feel good language that doesn't require a clear stance. How attractive is this membership option? When you ask questions like this, the results are vague and non-committal. They're easy to interpret in multiple ways, and usually as we see. They get easier to confirm or validate the thing that you wanted to validate in the first place.

Questions like this are really susceptible to molding responses into something that says, yes, we're doing a great job, right? They create false confidence in weak signals. For example, with this question, how attractive is this membership option? Even if I were to see five out of fives across the board in this one, I am not sold on the fact that this membership option that we're evaluating is really the best one.

Okay, so let's say, last example here are a co-working membership. I was at a WeWork yesterday and it was one of the most distracting places I've ever worked. It was a WeWork, like right in Midtown New York City. So this is the example I pulled. Let's say you want to test out this ad call to action for a me this UI element.

Maybe you're, there's a banner ad you're putting on a site. Maybe it's something on your own membership option site. But you want to test like the copy here and this summary with the high level benefits and pricing of this co-working space membership. Does this drive real interest? Does it just sound good?

Did we get the copyright, did we get the big benefits right on this one? And you're putting it out there to test. Here's the question we just looked at. How attractive is this membership option? This is a safe, subjective question that is guaranteed going to either lead to a ton of debate after, or unless everything comes back, as ones and twos, I'd say anything over a three here is going to get pushed in, get rounded up, right. To this is great. This is attractive. The problem is attractive. Doesn't is a safe, subjective word that doesn't require the participant to really commit to anything. It doesn't require them to relate it to their own experience.

It doesn't require them to put any buy-in or intent on something. It's easy to say something is attractive and it doesn't hold much weight. Here's a better question. Thinking about your own workspace needs, which best describes how you'd respond to this membership option? I'd look for more details to see if this fits my needs.

I'd consider it, but continue exploring other options. I'm not interested enough to look into this further. This isn't relevant to me. Again, maybe not the perfect question and a million different ways that you could word this, but the idea here is these results are going to be less susceptible to getting rounded up to interpretation.

They describe intent. They force the user to really think is this attractive or not? That's an easy one, right? You can just that. That doesn't mean much. You can just put. Pick a number for that. But if you see a question like this as a participant, you might come back and look at this, and say that price is cheaper than what I pay now.

I would want to see how many locations they have. Yeah. I'd consider it, but I continue continue exploring other options or that's a little expensive for me. I'd consider it. Follow up this with another question. What informed your answer, please explain. Get some of that information about how they made a decision.

This is going to be much richer feedback for you to go through and base decisions on, and it's going to be much less debatable, right? If someone were to say, I'm not interested enough to look into this further that I think is going to have, you're going to have a harder time rounding that up to, oh, this is great than you would with a 3.5.

So we have a combo question here of the misleading metrics and safe subjectivity. This one is easy to spot, but it's also one of the hardest to push back against, right? Because this hits on the main thing that a lot of you guys pointed out, at the top of this workshop, which is the fear you can't handle the truth.

A question like this one really puts it to clear specific evaluation of how this is working. So if you just want to, validate it and move on with your life and not actually care about how the, or if this is resonating a question like this that specifically and unde like undebatably gives you a response on that, that specifically where a user specifically saying what they would do next or how they would consider it and is not open to as much interpretation, that can be hard to get past. Especially if stakeholders or people who are you're working on that with are already like pushing things in a certain direction.

If you see that safe, subjective language again, one of the best ways to push back on it is to show the results or sample results and ask, is this enough to make a decision on? Is the fact that people are giving this a four out of five attractiveness, oh, this is an attractive membership option.

Is that enough for us to spend the next couple months building this? Does that give us a really clear signal that, yes, we nailed it. This is exactly what people want. This is the perfect membership option. So putting it out there like that and fast forwarding to the results is one of the most effective ways for all of these to put it out there.

Without necessarily like pushing back. But just letting that question sit and putting it up on the slides and saying, is this what we need to make a decision? Are we going to stake our next, sprint on people giving this a four out of five attractiveness?

Alright, feel free to drop some questions and I'm happy to get to those, but I'll do a some quick key takeaways with this.

One. When you're looking to identify or improve wishy-washy questions or wishy-washy research studies, one of the best things that you can do is run a pilot study, put it out there to five people, put it out there to just people in your, like even internally this can work. Get it so you can see those results.

And on Lyssna you have a results page that is really easy for anyone even well outside of the research department to see, and interpret and review. That is going to be the way to tie this stuff to reality, right? Eventually we are going to get responses for this, and we are going to have that meeting about what to do next.

Based on these responses, let's look at what these responses are going to look like before we do that and see if we need to tweak anything. Sammy, going back to your point there, define success upfront. This is getting out of the, let's see what happens model. Talk to your team, and this is a good follow up for this one here, right?

And say, how are we going to make this decision, right? What are we going to do when these results come back? What's our threshold? What are our next steps? You might not have it fully defined, you might. Get results back and say actually we need to do this. But if you have that conversation and align upfront, that is going to leave much less room for wishy-washy interpretation and, inaction and much more room and guidelines for what to do next.

Last one is commit. Now I probably don't need to tell you guys this 'cause you're researchers and product people, but this is the thing you got to, bring into these conversations, which is we're doing research to learn about what's working and what's not. That is the whole point of this to figure out what real people think, so we can use that to make our decisions.

If we are, There is no point of doing research if you're trying to sway it in a certain direction or water down these questions so that they'll be able to be interpreted however anyone wants. And it's, so obvious to us. But when you put it out there like that, it it really, I think, like simplifies everything and brings it back to the original goal of what we're trying to do.

You can even get it to the fact of like, why are we doing this in the first, why don't we just build, what if we're just going to build what we were going to build anyway? Let's not even do this research. Let's save ourself some time on it. Not the ideal path, but a question like that can help you commit and the people around you to really invest in we're saying we're going to build what, based on what our users need and the feedback that they're giving us. And we're going to ask them about it. Let's get the answers that we need and not dance around it or water things down.

Cool. I'm going to get to some questions, but first let me know if you guys enjoyed this workshop.

Hope it was an hour well spent for you. I'll hang out for a couple more minutes 'cause I see some good questions in the chat. But wanted to wrap up for anyone who has to go right now. Let me know if you enjoyed this and Let me know if you feel more. Confident as someone who does research than you were an hour ago I really hope that some of you, either immediately or in the next few weeks can go check out the research studies that you're running or ones that you're, planning with your team and make some of these changes that are going to have a major impact on that research, and a major impact on what comes out of it.

So if you're feeling like that that makes me happy and I'm seeing some good stuff in the chat here. Awesome. Thank you, Sammy. Here's the credits code. Let me put that back in the chat too so you guys can just click on it. Go claim that code that's 20 credits is enough to it depends on the study that you're running.

It's great for a pilot. So if you want to take any of the stuff that we did here today and run that pilot, show it to your stakeholders and say, Hey, Joe gave me 20 credits and told me to show you this results screen so we can fix these wishy-washy questions. That is a good starting point to do it.

Let's see some questions here. Is it possible to put a filter on a test to make sure evaluators have experience reviewing or hiring designers or design consultants?

Yeah, so in Lyssna there are lots of recruiting options and filters, Sharon, where you can target like different so many things. Everything from location to home ownership to employment status, role, all this stuff. With something like what you're asking about if they have experience reviewing or hiring designers, I would do that as a screener question.

So in Lyssna, you can create a screener question that will dis, that will either qualify or disqualify someone from participating in your study. So for your example, you might have a screener question that's like in the past, two years, have you hired designers or design consultants?

Have you evaluated designers or consultants with the intention of making a hiring decision? So for specific ones like that might not be in our panel filters you can, a screener question is the best way to do that. Next one. Tatiana, can you share how you would approach this question if the method was interviews?

Yeah. Let me go back to it here. So if the method was interviews, so generally with interviews, you would this becomes a lot easier, right? Because you're having that back and forth conversation and you're not forcing your participant into a specific. Answer format, right? So in an interview, like something like this, right?

Imagine saying to somebody, how attractive is this membership option? Wait, before you answer, give me a, all I need you to say is a number one through five, right? That doesn't really happen in interviews. For a question like this, like the safe subjectivity one, you want to make sure the wording is getting rid of that safe, subjective wording.

But there are a lot of aspects just in the natural flow of a conversation, in an interview that is going to even if you do ask a question that's safe or use this language, you're more likely to get that real behavior and have someone answer, give the, give a real answer rather than just directly just responding to exactly what you're saying.

So for this example you would change it to something like. Hey. Thinking about your own coworking space needs imagine seeing this offer here. Tell me a little bit about your level of interest based on this information, how relevant it is for you, something like that, to get them talking about their own experience and how they'd evaluate it.

But the key is still getting that word or those safe subjective words out of the equation, easier to do with interviews.

All right. Let me see if there's anything else since we're a little over here. Definitely. Does Lyssna provide video interviews? Yes. Yeah, Lyssna does provide video interviews, and that is like the pinnacle of getting real with your stakeholders, showing that video evidence. That's long before I, I used Lyssna, I was in iMovie cutting up long interviews for that nice 32nd soundbite that my stakeholders and clients could not ignore. So yes, those are extremely compelling. If you're trying to get stakeholders to, envision the results and Hey here's what, it's the responses that we're going to get.

Here's what's actual people are doing with our project. Here's a guy who's our ideal customer who can't figure out how to check out on our site. Extremely compelling. And yes, there are video interviews that you can get the recordings for and are shared on the results on the results page, so you can easily share them or edit them on your own to do it.

Thank you so much everybody. Awesome questions and great participation. I can see despite the wordy title of this workshop that the content is something you guys run into as well. So hopefully what we talked about here can help you improve that. Hopefully you got you some ways to put this into action.

Thanks so much for joining. I will send out the recording. You'll be getting that soon. And we got some really good workshops coming up soon, so keep an eye out for that and hopefully I will see you in one in the next couple weeks or months. Have a great day everybody, and see you soon. Thank you.