How the Survey Shapes the Game (Patreon)

Do your opinions change the game? You betcha. Let’s talk about how and why.

Hey folks, James here. Ever since we put out the first MCDM RPG Patreon playtest packet and survey, I’ve gotten a lot of really great questions from folks about how we analyze your feedback, how we put it into action, and how much it matters. As the person who creates and reads the responses to most of the MCDM game design surveys around here, allow me to pull back the curtain for you all!

Survey Deadline: January 23rd at 11AM Pacific US

Before we dive into the good stuff, let me remind you that you have until January 23rd at 11AM Pacific/2PM Eastern US time to fill out the survey. Please only fill out the survey once. Feel free to send it to anyone who playtested the MCDM RPG, even if they’re not patrons. However, please don’t post the survey or packet anywhere publicly. The playtest packet will remain attached to this linked post so you can continue to play, but the survey closes on January 23rd. I’m not sure when the next packet will be coming your way, but it’ll probably be a few months as we review and implement your feedback and expand the game.

All right, with that bit of admin out of the way, it’s time for … oh, another bit of admin.

Connect Your Discord and Patreon Accounts

If you’re a member of the MCDM Discord channel, you should link your Patreon and Discord accounts. You can read about how to do so here. Linking your accounts will give you access to our #mcdm_rpg-patrons channel on Discord. It’s an incredibly popular place for patrons to discuss the MCDM RPG and the latest playtest with each other. Tons of great tips, ideas, and conversation await!

It’s time to talk about surveys!

How We Build Surveys

When I create a survey for MCDM, I’ve got a few different goals in mind. First, I want to create clear, objective questions. The purpose of the survey is to find out what you, the patrons, do and don’t like about the game as it currently stands. I’m interested in your honest opinions, not having praise heaped upon our design. The survey asks questions like, “How satisfied are you with an attack's damage automatically being applied to a target?” instead of “Did you miss the attack roll at all?” The first is a question asking for your honest opinion without leading you toward a particular answer. The second assumes you’ve played a game with an attack roll and never one without and leads you toward a positive response. I genuinely want to know how satisfied you are, so that’s what I’m going to ask.

I also mainly include two kinds of questions. The first are answered with a 1-to-5 Likert scale. For instance, the survey asks, “On a scale of 1 to 5, how satisfied are you with the MCDM RPG overall?” 1 is “not at all satisfied” and 5 is “very satisfied.” These questions help us get a broad view of how folks feel about the MCDM RPG and its many rules. If I want to immediately know how many people are satisfied with an aspect of the game, like the Notice skill, for instance, I can quickly scroll to the result of that question and get an instant breakdown. These kinds of questions also make it easy for a busy person who doesn’t want to write a lot of specific details to still give us feedback. It doesn’t take too long to select the number that matches your opinion and move on.

However, we also want people with the time and inclination to write their opinions to get specific in their feedback, so I also write a lot of open-ended questions, such as, “Anything else you want to tell us about the rules for negotiation?” It’s never required that a person fill these out, but we do value these responses.

I also don’t want to ask too much of the person filling the survey out. While this first survey is long, we kept to questions that we really wanted to ask at this stage. For instance, we didn’t ask a lot of questions about layout or creature statistics, because we know those things are going to change quite a bit. We also don’t ask the same question in a series of different ways.

Feedback We Want

One question I get a lot is, “What kind of feedback do you want in a survey?” For us, the best feedback comes not in the form of suggestions but rather as criticism. We want to know what you like and dislike. Leave the actual fixing of the problem to us. To metaphor it, your plumber just expects you to show them where your leaky pipes are. They don’t expect you to tell them how to fix the pipes. We’re the plumber! The rules you don’t like are the leaky pipes.

A lot of folks might wonder why we feel this way. When a person leaves a suggestion, they don’t always tell us what problem they’re having with the game that the suggestion is trying to solve. “You should have rules for super secret motivations in negotiation that work like this…” followed by seven paragraphs of original negotiation rules doesn’t really help us. Saying, “My players were frustrated by the rules for negotiation because they felt like they kept hitting dead ends with it,” tells us so much more.

We also tend to get a lot of suggestions that are things we’ve already tried. We know that armor as damage reduction doesn’t work. Trust us, we already went down that road! We don’t want you to waste your time suggesting it.

Finally, game design is work. Don’t give us your good ideas for free. Keep those to yourself. You’ll go much further that way!

Analyzing Feedback

Analyzing feedback starts with those Likert scale questions. I go through and look at the average scores. Ideally, we want 4 or better in a score. That means people really like it! I take notes on these averages and identify the parts of the game that scored the lowest as problem areas that require action, though I don’t know what action I’ll be taking yet.

I also note the things people really seem to like. Everything in this game is connected, so changing one rule could really change another that has a high score. That’s actually something we do all the time. Sometimes a game is made better by changing or losing a popular feature (like having stamina AND health or surges AND heroic resources in the early days … people loved that stuff but the game runs smoother for the changes). But it is good to note which features are popular with people, because those reveal a lot about what our audience likes and we’ll also now be aware if we do change something beloved, so we can take what folks liked about it and put it somewhere else in the game. For instance, even though we got rid of surges, we were able to incorporate what folks like about them into heroic resources.

After I look at the numbers, I dive into the comments. This is a long process, and I tend to do it survey by survey. It gives me good context to read all of a person’s comments at once, instead of reading everyone’s comments about skills and then combat and then negotiation for instance. I already have a top-level view of the prevailing opinion on matters thanks to the Likert scale numbers. I don’t need to read comments by category. Reading them by person allows me to better understand what the individual is trying to say. For instance, if someone has a problem with the way Victories work, their criticism of Victories may be in every comment box, because Victories interact with a good portion of the game. I can see based on that single survey that this is a problem one person in particular has and not a bunch of people commenting about it in a single place. (By the way, even if one person has a comment on something everyone else loves, we might still address it. More on that below.) Folks who leave comments will also sometimes refer to answers on previous questions, so looking at results survey-by-survey really helps with that context too.

As I go over the answers, I pay particular attention to the comments that apply to the low and high scoring areas of the game. It gives me insight into why people don’t like or like something. For instance, if the rules for Recoveries scored low and there are a lot of comments saying, “The game was easy because heroes had so many Recoveries,” then it’s pretty clear-cut that we should reduce the number or effectiveness of Recoveries. The numbers and the comments agree! When that happens, my job is easy.

But I read all the comments—all of ‘em. While majority opinion is helpful in shaping the game, it’s not what drives the game’s design. The design department at MCDM (Matt, myself, and folks we hire to work on the game) have the final say. We have very clear design goals for this game. You’ve probably heard us say “tactical, cinematic, heroic fantasy” about a zillion times now. Matt and I have opinions about where the game could be improved, even for rules items that are scoring high. We know from experience that you all do too, and sometimes we agree with those opinions. In fact, one person might leave a comment on a feature everyone else loves, but when I read that single comment it’ll make me say, “This person is right. We need to change this.” Ultimately, the opinion of the MCDM design team is what drives the game and causes the change. We feel that any piece of feedback could have a nugget of criticism that could help us improve the game, so I read it all.

Side note, when I read playtest feedback, I check my ego at the door. It’s a humbling experience, but it makes the game better. I try to remove myself from it and be as objective as possible. It’s not always easy, especially when a game is personal to me like this one, but I always remember I’m here to help make the coolest and best game I can. It’s not about me. It’s about the experience folks have playing it at the table. Thank you for being constructive in your feedback! It makes my job that much easier.

After I read the surveys, I have a meeting with Matt to talk about where the game could be improved and we brainstorm solutions together. He also pokes around the feedback to see what folks are saying, but right now it’s my job as lead designer to dig deep into that feedback. After that meeting, I get revising!

That’s the basic process we go through! I hope that helps answer your questions. Looking forward to seeing what you think of this game.

Ex animo,

James Introcaso

MCDM Lead Game Designer