The content is essentially the same. Instead of typing the keywords you click on hyperlinks that are highlighted in the text and shown in a separate list next to the story. A clickable, dynamic map of locations is displayed on the opposite side. There's background music, but sound support is still a bit shaky in some browsers. Internet Explorer 7 or earlier will not work.

It uses the yet-unpublished version 1.2 of Vorple and demonstrates the use of the button interface in the map and in the keyword list, and tooltips that are displayed as brief instructions and as labels for the map. Under the hood it uses disposable links and other similar features. It's also using the IF Recorder plugin, probably the first time for an Undum story.

• Play online
• Source code

Looking at the data from before and after the update, here's what happened to the total amount of invalid commands:

In addition to new synonyms the update includes a new class of error message. The earlier version responded with "That is not a keyword the story recognizes" to all invalid input. The update shows the same message when the player gives only one word as input, but if the player had given more than one word the error message is ”Please type a single keyword to advance the story” followed by a list of available keywords.

This was inspired by the different ways people had given invalid input: some gave traditional IF commands, some typed more than one keyword at the same time. In addition to the new error message giving more accurate instructions on how to interact with the story, it also explicitly lists the input it accepts, leaving no room for confusion. I looked at how this affected "ragequitting" after giving invalid multiword input during the first turn:

(The sample set is so small that the results fit well within error margins, but they are still nice numbers to look at.)

In addition to reducing the total number of invalid input, the amount of error messages per game dropped as well. In the previous version about half of the players didn't encounter any error messages while playing, but more than a quarter bumped into them more than once, 10 percent more than three times. In the latest release the no errors group jumped up to two thirds and those who saw more than one error message dropped to less than 10 percent.

It was expected that by making the parser more error tolerant and helpful the percentage of people who played the story to completion would go up, as well as time spent with the game, measured here with the average number of turns per game (not counting those who quit without typing a single command).

As the graphs show, the percentage of people who quit after one turn or right after the intro doesn't show any significant change. There are people who don't find this kind of medium their thing and there's not much that can be done to make them keep playing. in the other end of the scale are the hardcore IF fans who play anything they can get their hands on. Most players fall somewhere in between and to keep them engaged requires both a good story and smooth gameplay.

Betatesting can do a lot, but analysis of real life transcripts can reveal surprising things that have a significant effect on how players enjoy the game. Who would've thought that invalid one-word and multiword input should have different error messages? What would a similar stack of transcripts from games with full parsers reveal?

The good news is that I've almost finished with making the transcript recording functionality an actual Parchment plugin. It'll be released soon, first with Z-machine support and later with Glulx support, and even later with statistical tools to make all kinds of graphs like the ones presented here.

Last Wednesday however C.E.J. Pacian wrote a blog post about it which was picked up by the IndieGames.com blog. From there a couple of people tweeted about it and suddenly the play count jumped to about 600 plays per day. The main Parchment site reported similar amounts of traffic.

I'm running the online version of the story on a modified version of Parchment that saves the transcripts to the server. I now have 1557 transcripts consisting of 23 896 turns (and counting, but the traffic is slowing down considerably). To my knowledge this is more play data collected than for any other single IF game so far.

So here's a random bunch of analysis from those transcripts. There are no spoilers other than keywords used throughout the game, but if you haven't played yet the data would probably make more sense if you at least checked out a couple of turns just to see what the parser looks like. Online version and downloads are available.

Parser errors

Since the gameplay is just typing single keywords given by the story, the amount of understood input was substantially high (93%). Most of the errors were typos (53.3% of all invalid input).

Standard IF commands were disabled so they count as unrecognized commands. The "sign of confusion" category contains commands like HUH, ?, WHERE, or verb commands that don't follow the standard IF convention (I GO OUT). A "poke" is the player testing the limits of the parser by trying to communicate with it (HELLO, BYE, OH NO) or trying to do things out of character, mostly involving trying to kill poor Kevin (DIE, KILL COUSIN), and even trying if the prompt recognizes Unix commands (LS, CD ..). "Gibberish" is a random string of letters or arbitrary words that don't appear anywhere in the story text (WALRUS, AWESOME). Non-keywords are words that appear in the story text but aren't keywords that you can choose (MOM, HOME).

Sticking with it to the end

About 11% of players quit without typing a single command, the same amount after the first command and a quarter kept playing until the game's conclusion.

What happens when the story tells the player that the input wasn't understood? In this case the game tells up front what kind of input it accepts. The parser can complain about the input only if the player disregards or misunderstands the instructions, makes a typo, or if the game has a bug that doesn't parse valid input correctly. There is a bug in Starborn where commanding MAGELLAN in one room gives a frustrating "Which do you mean, Magellan or the Magellan?" disambiguation question, but other than that the keyword system is immune to implementation issues where input consistent with the game's rules is disregarded because the author didn't implement a correct response.

Player made errors like typos can affect the player's experience just as much as the game's shortcomings. If you aren't able to make progress in the game, no matter the reason, you're likely to become frustrated and quit. I took a look at what happened when the player gave invalid input at the first turn. I expected that those players would be more likely to quit immediately than those who gave a valid command, but it turned out that there was no significant difference in that regard. They were still more likely to stop playing later on before reaching the end.

The reason for this might be that people who are prone to making typos are more likely to become frustrated when the story keeps stalling. This theory is supported by the fact that people who didn't complete the story gave almost twice as much invalid commands than people who completed the story (proportionate to the total amount of input; transcripts with less than 10 moves not counted).

The first turn

The story starts with short instructions on how to play.

After pressing space the screen is cleared and the game starts.

At this point the player has four keywords to choose from. It was expected that as the first command the first keyword was picked the most often and the last keyword the second most often and the transcripts show that the assumption was correct.

What wasn't as expected was that some people (20 out of the total, that's more than one percent) typed every available keyword at the same time ("MUSIC COUSIN WEIGHTLESSNESS HALLWAY"). Other typical mistakes were typos and assuming that the keyword in "...my COUSIN's sleep" is "COUSIN'S" including the apostrophe and the "s". Presumably out of habit the players would also try the usual IF commands (X ME, TURN MUSIC ON).

Category "other" includes, among others, choosing words from the text that weren't keywords, swearing, and trying to communicate with the parser (HI, OK, LOL).

Leading the player

I assumed that if choosing a keyword leads to a description that contains exactly one new keyword, the player is likely to pick that keyword instead of any of the previous yet unexplored ones. For example, if the player chooses the keyword MUSIC in the intro, the description will be:

The transcripts reveal that the most common command was indeed MAGELLAN, although I expected the figure to be much higher.

By default the story shows available keywords in the story text in upper case, so about one sixth of the players typed them in upper case as well.

("Other" includes empty input, numbers, and other input where there are no characters.)

As the game never instructs the player that they can type only one of the words in a two-word keyword, it wasn't surprising that the players were generally inclined to type the entire name of the keyword.

Typos

How many ways can you write WEIGHTLESSNESS? At least 20 different ways:

Because you can never take into account every possible way a word could be typoed, it'd be great if the parser could just go through the available vocabulary and choose the closest match to the player's input. You could then have the parser understand the most outrageous misspellings the players could come up with. This game is an exception of sorts because all available keywords (and therefore all acceptable commands) are at all times visible to the player so there's no need to be wary of the parser choosing a word that the player hasn't encountered yet, but even in a traditional IF game you could have the parser go through only the words that have been previously shown to the player.

Further steps

Just by fixing the 30 most common unrecognized commands fixes one third of all unrecognized input. The cases where the player types more than one keyword at a time or tries to use a standard IF command can be handled with a better error message, something like "Please type keywords only, one at a time. Currently available keywords are ..."

Analyzing the transcripts is great for finding out which unimplemented commands need attention or what synonyms to add, but it's by no means a replacement for traditional testing: the transcripts don't tell you how good the story is, do the puzzles work, is the pacing correct, are there spelling errors in the text... The benefit of data collection from a live release is that the data represents the actual audience. The beta testers are probably on average more experienced players than people playing the game after its release.

It'd be great to have similar volume of data from a traditional parser IF game so that the results would be more generally applicable. Even a couple of transcripts from a lot of different games would probably show interesting results.

(Updates: follow-up after an update; the recording plugin for Parchment released)