Jeff Carlsen over at Apathy Games posted this article, which I found insightful. The point I was most excited by is that there's no agreed upon metric for RPG analysis; I also understand that everyone is out there trying to create their own experience, but once you have two thousand pretty intelligent pudgy white guys talking about the relative merits of their opinions you tend to get into trouble.
- Fun: I always have fun with my games, but the kind of fun I have is different from what other groups are doing. We can safely count out "how much fun did you have" as a quantitative measurement of a game's worth.
- Roleplaying: Again, different groups get into this in different ways. We shouldn't slam a system for our own inability to roleplay - grognards, I am looking at you.
- Do the pregens have the ability to get through the adventure? I know for one I'm always alarmed when there's not a single character with more than a d6 in combat skills. Call me crazy, but I like the tactical element myself; even if you aren't so into combat, you have to concede it's more fun to win a fight on the tabletop than to lose.
- Is the system an impediment to the story? This is a tricky one but I think it's worth mentioning. If you spend so much time looking up the rules that you forget what you're doing, then either the adventure needs work (calling up a bunch of esoteric seldom-used rules as it does) or it's not meant for the system.
- Is the story any good? Does it have a beginning, middle, and end; does it have a clear villain; does it have a clear reason for the PCs to be involved; does it change the world of the game in some way? As a player, I want to feel like my character matters. Some of these ideas go into my own bias about how world and adventures should be constructed, so I'm open to feedback.
- Is there a reason to drive forward? What's keeping the PCs from throwing up their hands and saying "nope, this can't be done?" What is at stake?