Ben has an interesting post: “Stop Playtesting”. More specifically, he’s targetting where playtesting is actually toxic to a design process.
Rebuilding This Warp Drive In Mid-Flight
So thinking about Spacerpunk: it has evolved in a decent fashion as I put the game in front of good play groups, and it has changed significantly in response to the group’s feedback. However, caveats to this:
(1) I have come to carefully filter player feedback to tell the difference from core problems with my game vision (that I should change), vs personal preferences or mismatched visions (which I should probably not change in the game itself, but certainly could drift in an actual game to suit the group; but this doesn’t get me further in terms of design).
(2) I try not to immediately jump on feedback until it has recurred in a multisession pattern. (Or unless it is so catastrophic and clear to be unavoidable.)
(3) Interestingly, my resolution mechanics have been all over the place during the design of this game, but the structure of the setting and player situation has been pretty steady. I suspect that the fluff of my game is the important part!
It’s obvious, though, that playtesting mid-design can result in jittery, mixed feedback from players who may not share your interests or may simply have a skewed experience at the table. Scrambling at the whim of every respondent quickly gets you to Design By Committee and, thus Design Hell; I get there easily enough on my own.
Drinking My Own Kool-Aid
I think a flaw in my game design abilities – perhaps due to a relatively unrefined process? – is that I’m not the best predictor of how a system or mechanic will work in a social context. I easily get caught up in the narrative of what I’d like the system to do, and do not predict what is the more obvious course of action.
A key example of this exists in The Dance and The Dawn. I initially had an economy of Ice Tokens where players were rewarded for being cruel by sabotaging other players (but at the cost of arming them with the very resource they had just used).
I imagined this could have lead to some delightfully escalating bouts of sabotage with a sort of prisoner’s dilemma, but in practice the currency did not motivate well. There was so little incentive to attack other players – and a near guarantee of retribution – that abstaining from the Ice Tokens was the most obvious strategy. Players only sought out Ice Tokens if they wanted to engage in sabotage for aesthetic reasons, rather than competitive ones.
I’d like to think I’ve gotten better at this kind of objective analysis, but I think my playstyle also impacts my ability to playtest. I instinctively drift a game to fit the style of a group and engage with mechanics on a somewhat emotional/fictional level, rather than trying to optimize my play. (Ex: suboptimal raises in Dogs in the Vineyard to reflect my fictional intent, rather than playing to win and creating suboptimal fiction.) When I’m not more stringent than this, the result is brittle design.
math is hard
I agree that playtesting is an expensive place to figure out how mathematical probabilities work out. I wish it was easier for layfolk to figure out probabilities – hell, I generally just write a script to generate results naively and take the average – but it’s still vital. If you have a rule that’s “keep doing X until you roll doubles”, and you don’t know whether that averages to 2 die roll or 12, then your design is missing something huge.
Math doesn’t have to come first of course, and thinking that way probably doesn’t work for most designers. If you want to defer on the probability work, design your game and, once it has taken shape, take note of what kinds of probabilities you hope for each mechanic to produce. (Is it more like 50/50 or a sure thing? Do you steadily get better, or does your chance of success cap out somewhere?) You can doublecheck the actually produced probabilities, and where necessary work backward from desired probability to a mechanic that will work.
(Hey readers: what tools are there for figuring out math probabilities in an easily accessible way? I write stuff in Ruby, but that’s rather niche.)
ETA: Continued in part two.