The games are placed in the chart with horizontal axis showing the IFComp score and vertical axis showing the IFDB score. Hovering the mouse over the dot displays the title. The blue diagonal line is the trendline. If the dot is above the trendline, it means it has been ranked higher in IFDB than in IFComp (in relation to other games in the same comp). Conversely if the dot is below the trendline the game did better in the comp than in IFDB. The distance from the line tells how big the disrepancy is.

Only games with at least 5 votes in IFDB are counted. That includes everything from 2008 on. Older bottom-tier games tend to have fewer votes. The only disqualified highly-ranked game is A New Life with only 4 votes which tied for second place in 2005.

(The charts might not show up if you're reading this from the RSS feed or Planet IF.)

If we look at how IFComp winners are rated in IFDB compared to games released the same year we notice that the best games aren't always found from the competition. In fact based on IFDB ratings the comp winner is the best-rated game of the year in only two cases, and more than half the time the comp winner isn't even the best rated IFComp game in IFDB.

The XYZZY award for the best game of the year goes to an IFComp winner roughly one third of the time, and one third of the time it isn't even nominated. Only Lost Pig in 2007 hit the jackpot (IFComp winner, highest rated game of the year, XYZZY award).

In the table below the first number column is the ranking among the competition games and the second is the ranking among all games that year. The last column shows if the game was nominated or won the XYZZY award for best game. For example in 1996 The Meteor, The Stone And A Long Glass Of Sherbet won the comp, was nominated for XYZZY and in IFDB is rated the 5th best comp game and the 7th best game of 1996.

(Data retrieved 2013-11-21.)

Here are the "best of"s of each year. The number in the "IFDB best of comp" column is that game's actual placing in IFComp that year.

(Blue Lacuna is listed as a 2008 game in IFDB, but it was eligible for XYZZYs in 2009. If you count Blue Lacuna as a 2009 game then Violet is the highest rated in 2008.)

What's obvious here is that people have different criteria for rating games in IFComp and IFDB. Lists and Lists, which teaches how to program in Scheme, is technically impressive but hardly something that people would want to win the IFComp.

There's also an inherent bias with this kind of scoring where you obviously need to play the game before you can rank it. When you're participating in the IFComp as a judge you often play the games without knowing anything about them beforehand, or play and rate them all even if you do. Outside the comp you're likely to not even pick up a genre that doesn't interest you; if roguelikes are not your thing you're unlikely to play and rate Kerkerkruip.

Moral of the story? Firstly, even if your game doesn't rank well in IFComp it might still be highly appreciated in other contexts or as time passes. Secondly, releasing in IFComp is not (and has never been) required for a good game to get the appreciation it deserves.

And finally here are the best IFComp games ever according to IFDB.

When we talk about development systems in IFDB, it's good to remember that some systems have better coverage than others. Parallel communities like Quest (201 games in its own site, 26 in IFDB), non-parser systems and the AIF crowd are underrepresented. Another factor that skews statistics is that game listings tend to display the system in which the game is available now—for example Scott Adams adventures are listed as Inform 6 games because the downloads are for the Inform ports.

In the all-time popularity chart Inform 6 is the clear leader: a quarter of all games have been made with it. Inform 6 and 7 together cover more than a third of all games.

The chart for current (2010-2012) systems is a bit different. Inform 7 has risen to dominate the field with the market share of two thirds. Inform 6 has dropped significantly and all other systems are in the margins. Does this mean that the field has become less diverse or that IFDB has not kept up with new systems?

The Commercial Era

I've split the statistics in two eras: the commercial era (before 1994) and the hobbyist era (from 1994 on). 1994 marks the beginning of a renaissance—IF died commercially but at the same time the publication of Inform practically created the modern hobbyist scene.

Many of the commercial era games don't have the system marked down or have a suspicious "none" (were they written directly into machine code?) or a vague "custom". The most popular system was Eamon, which was exclusively used to create stories in the Eamon game world. The Quill was almost as popular, although the Wikipedia article says there were more than 450 commercial The Quill games (203 listed in IFDB) which would make it more popular than Eamon.

The large number of Inform 6 games is because of ports, as mentioned above.

Looking at the numbers by year, Eamon reaches quickly its highest point in mid-80s and declines from there more or less steadily. The Quill and AGT have a more steady growth all the way to the 90s. As you might imagine, BASICs popularity drops quickly after dedicated development systems become available.

Here's the same chart but with percentages from the total number of games: the vertical thickness of the area denotes more games in relation to other systems, the full height being 100%. The data in this chart goes up to 1998 so that we see how all the "old" systems drop practically to zero in popularity by then.

The Hobbyist Era

In the mid-90s the playfield changes completely. TADS, Inform, ADRIFT and other systems are released and especially Inform gains popularity.

Inform's domination is apparent in the percentual view. TADS 2 has steady popularity until the turn of the millennium when it begins a downward slope. TADS 3 gains a small foothold but never grows very much. Hugo has seen a small resurrection lately, mostly thanks to a small but active group.

Inform

After Inform 6 was released, it rose to cover almost half of all published games in only two years. The same happened when Inform 7 was released: in a few years it took the lead, eating mostly Inform 6's popularity.

This year only a handful of Inform 6 games have been released and Inform is at almost 70%.

The question is: has Inform 7 attracted people into IF who would otherwise not have done so, or would people who now use Inform 7 started to use something else if it had never been released? Surely there are people in both groups, but the graph above suggests that Inform 7 is eating away the Inform 6 userbase, not others. The "everything else" line does not seem to have any correlation with the Inform 7 line.

TADS

TADS 2 was at its height in 1999 when 27% of all games were written using it. Another peak was in 2002 after TADS 3 was released, but since then their popularity has been on a steady downward curve.

Again, the statistics show no correlation between Inform and TADS. Peaks in Inform's popularity do not show anywhere in the TADS statistics. (You could argue that I7 might have cut short TADS 3's slow climb between 2004 and 2008, but I very much doubt it.)

A few months back Mike Roberts, the curator of IFDB, together with Andrew Plotkin released the full IFDB database dump. This is certainly a great deed, since the IFDB is the most complete source of information about IF to date and there's always a risk of data loss if the database is behind only one person. The database is an invaluable resource to (future) historians, researchers and people who love statistics.

And I do love statistics.

I've pulled a lot of interesting graphs from the database and will present them in a series of blog posts, of which this is the first one. We'll start slow by looking at the games' publishing dates and progress later to searching for more and more specific correlations.

It's worth noting that while IFDB has a lot of information, it's by no means comprehensive or 100% accurate. Therefore all this data reflects IFDB contents rather than the real world. How much the truth differs from the available data is open to discussion.

Click on the images to see the full data set, an interactive chart and chart display options. All available statistics can be seen collected on this page. More graphs are added as the series progresses.

You can also gladly suggest in the comments what kind of data you'd like to see analyzed.

Publishing date by year

There are 4173 games in the database, of which 4027 have a known publishing date. The graph for total number of games over time looks like this:

More informative is the number of games published each year:

You can see the "golden age" from mid-80s to early 90s, a small dip, and a steep rise starting soon after. I have no explanation for why there were so many releases in 2001–2003. In 2001 there were twice as many releases as in 2009.

In recent years the number of releases has varied between 115 (in 2009) and 173 (in 2010). There's no discernible trend to either direction and there's not enough data to make any predictions about the future. 2012 is not included in the graph but at the end of October there were 164 releases which is already more than in 2011 (153 releases).

Publishing month

This (and the next graph) is wildly inaccurate since about 70% of games have only the publishing year listed, not the exact date (which is why January 1st is excluded). Still, you see what you'd expect: IFComp games are released in October so it has the most releases and November is not a good time to release because the competition draws all the attention.

Publishing weekday

Games are released most often on Sunday and least often on Friday. Shouldn't be a surprise that weekends are the most common release days as most games are hobbyist efforts.

Next time we'll look at development system popularity.

The plugin saves more than just plain text: it preserves text formatting and saves status lines as well, so the transcripts look (almost) the same as the actual game. (See this example of Bronze with status lines, colors and text formatting. Note that Parchment uses small caps in place of bold text.) The saved transcripts can be viewed with or without the status lines. In addition when the game waits for a single keypress, non-alphabet characters like enter, space and arrow keys are marked in the transcript.

There's a live demo with a transcript viewer available. Games played in the demo installation can be seen in the viewer.

The plugin and accompanying tools can be downloaded from Google Code where you'll also find installation instructions. Only Z-machine is supported at the moment, Glulx support will be added later.

Looking at the data from before and after the update, here's what happened to the total amount of invalid commands:

In addition to new synonyms the update includes a new class of error message. The earlier version responded with "That is not a keyword the story recognizes" to all invalid input. The update shows the same message when the player gives only one word as input, but if the player had given more than one word the error message is ”Please type a single keyword to advance the story” followed by a list of available keywords.

This was inspired by the different ways people had given invalid input: some gave traditional IF commands, some typed more than one keyword at the same time. In addition to the new error message giving more accurate instructions on how to interact with the story, it also explicitly lists the input it accepts, leaving no room for confusion. I looked at how this affected "ragequitting" after giving invalid multiword input during the first turn:

(The sample set is so small that the results fit well within error margins, but they are still nice numbers to look at.)

In addition to reducing the total number of invalid input, the amount of error messages per game dropped as well. In the previous version about half of the players didn't encounter any error messages while playing, but more than a quarter bumped into them more than once, 10 percent more than three times. In the latest release the no errors group jumped up to two thirds and those who saw more than one error message dropped to less than 10 percent.

It was expected that by making the parser more error tolerant and helpful the percentage of people who played the story to completion would go up, as well as time spent with the game, measured here with the average number of turns per game (not counting those who quit without typing a single command).

As the graphs show, the percentage of people who quit after one turn or right after the intro doesn't show any significant change. There are people who don't find this kind of medium their thing and there's not much that can be done to make them keep playing. in the other end of the scale are the hardcore IF fans who play anything they can get their hands on. Most players fall somewhere in between and to keep them engaged requires both a good story and smooth gameplay.

Betatesting can do a lot, but analysis of real life transcripts can reveal surprising things that have a significant effect on how players enjoy the game. Who would've thought that invalid one-word and multiword input should have different error messages? What would a similar stack of transcripts from games with full parsers reveal?

The good news is that I've almost finished with making the transcript recording functionality an actual Parchment plugin. It'll be released soon, first with Z-machine support and later with Glulx support, and even later with statistical tools to make all kinds of graphs like the ones presented here.

Last Wednesday however C.E.J. Pacian wrote a blog post about it which was picked up by the IndieGames.com blog. From there a couple of people tweeted about it and suddenly the play count jumped to about 600 plays per day. The main Parchment site reported similar amounts of traffic.

I'm running the online version of the story on a modified version of Parchment that saves the transcripts to the server. I now have 1557 transcripts consisting of 23 896 turns (and counting, but the traffic is slowing down considerably). To my knowledge this is more play data collected than for any other single IF game so far.

So here's a random bunch of analysis from those transcripts. There are no spoilers other than keywords used throughout the game, but if you haven't played yet the data would probably make more sense if you at least checked out a couple of turns just to see what the parser looks like. Online version and downloads are available.

Parser errors

Since the gameplay is just typing single keywords given by the story, the amount of understood input was substantially high (93%). Most of the errors were typos (53.3% of all invalid input).

Standard IF commands were disabled so they count as unrecognized commands. The "sign of confusion" category contains commands like HUH, ?, WHERE, or verb commands that don't follow the standard IF convention (I GO OUT). A "poke" is the player testing the limits of the parser by trying to communicate with it (HELLO, BYE, OH NO) or trying to do things out of character, mostly involving trying to kill poor Kevin (DIE, KILL COUSIN), and even trying if the prompt recognizes Unix commands (LS, CD ..). "Gibberish" is a random string of letters or arbitrary words that don't appear anywhere in the story text (WALRUS, AWESOME). Non-keywords are words that appear in the story text but aren't keywords that you can choose (MOM, HOME).

Sticking with it to the end

About 11% of players quit without typing a single command, the same amount after the first command and a quarter kept playing until the game's conclusion.

What happens when the story tells the player that the input wasn't understood? In this case the game tells up front what kind of input it accepts. The parser can complain about the input only if the player disregards or misunderstands the instructions, makes a typo, or if the game has a bug that doesn't parse valid input correctly. There is a bug in Starborn where commanding MAGELLAN in one room gives a frustrating "Which do you mean, Magellan or the Magellan?" disambiguation question, but other than that the keyword system is immune to implementation issues where input consistent with the game's rules is disregarded because the author didn't implement a correct response.

Player made errors like typos can affect the player's experience just as much as the game's shortcomings. If you aren't able to make progress in the game, no matter the reason, you're likely to become frustrated and quit. I took a look at what happened when the player gave invalid input at the first turn. I expected that those players would be more likely to quit immediately than those who gave a valid command, but it turned out that there was no significant difference in that regard. They were still more likely to stop playing later on before reaching the end.

The reason for this might be that people who are prone to making typos are more likely to become frustrated when the story keeps stalling. This theory is supported by the fact that people who didn't complete the story gave almost twice as much invalid commands than people who completed the story (proportionate to the total amount of input; transcripts with less than 10 moves not counted).

The first turn

The story starts with short instructions on how to play.

After pressing space the screen is cleared and the game starts.

At this point the player has four keywords to choose from. It was expected that as the first command the first keyword was picked the most often and the last keyword the second most often and the transcripts show that the assumption was correct.

What wasn't as expected was that some people (20 out of the total, that's more than one percent) typed every available keyword at the same time ("MUSIC COUSIN WEIGHTLESSNESS HALLWAY"). Other typical mistakes were typos and assuming that the keyword in "...my COUSIN's sleep" is "COUSIN'S" including the apostrophe and the "s". Presumably out of habit the players would also try the usual IF commands (X ME, TURN MUSIC ON).

Category "other" includes, among others, choosing words from the text that weren't keywords, swearing, and trying to communicate with the parser (HI, OK, LOL).

Leading the player

I assumed that if choosing a keyword leads to a description that contains exactly one new keyword, the player is likely to pick that keyword instead of any of the previous yet unexplored ones. For example, if the player chooses the keyword MUSIC in the intro, the description will be:

The transcripts reveal that the most common command was indeed MAGELLAN, although I expected the figure to be much higher.

By default the story shows available keywords in the story text in upper case, so about one sixth of the players typed them in upper case as well.

("Other" includes empty input, numbers, and other input where there are no characters.)

As the game never instructs the player that they can type only one of the words in a two-word keyword, it wasn't surprising that the players were generally inclined to type the entire name of the keyword.

Typos

How many ways can you write WEIGHTLESSNESS? At least 20 different ways:

Because you can never take into account every possible way a word could be typoed, it'd be great if the parser could just go through the available vocabulary and choose the closest match to the player's input. You could then have the parser understand the most outrageous misspellings the players could come up with. This game is an exception of sorts because all available keywords (and therefore all acceptable commands) are at all times visible to the player so there's no need to be wary of the parser choosing a word that the player hasn't encountered yet, but even in a traditional IF game you could have the parser go through only the words that have been previously shown to the player.

Further steps

Just by fixing the 30 most common unrecognized commands fixes one third of all unrecognized input. The cases where the player types more than one keyword at a time or tries to use a standard IF command can be handled with a better error message, something like "Please type keywords only, one at a time. Currently available keywords are ..."

Analyzing the transcripts is great for finding out which unimplemented commands need attention or what synonyms to add, but it's by no means a replacement for traditional testing: the transcripts don't tell you how good the story is, do the puzzles work, is the pacing correct, are there spelling errors in the text... The benefit of data collection from a live release is that the data represents the actual audience. The beta testers are probably on average more experienced players than people playing the game after its release.

It'd be great to have similar volume of data from a traditional parser IF game so that the results would be more generally applicable. Even a couple of transcripts from a lot of different games would probably show interesting results.

(Updates: follow-up after an update; the recording plugin for Parchment released)

Try it out yourself: Click here to start a game and then open this page to a new browser window. Enter a command in the game and see how the transcript in the live feed updates. (If you don't see your game in the live feed page, reload it. It shows the three latest games being played at the time the feed page was loaded.)

And then there's the beef: I've made a rudimentary statistics page that shows some interesting information using the collected data, including most used commands and average turn count and playtime.

The obvious use for this is of course collecting information on how people play the game and using that information to make improvements for the next release. You could also use this for closed beta-testing -- give the web address to your testers and Parchment saves the transcripts automatically. The testers don't have to remember to start the transcript or send it to the author.

Statistics from other games

First I want to stress that at the moment this is just a demo of a hack. It can and will behave strangely especially if the game bypasses the standard parser and takes input directly (menus etc.) That said, if you want to try other games than the sample game, you can use either http://parchment.game-testing.org/demo/parchment.full.html?story=xxx or http://parchment.game-testing.org/play/parchment.full.html?story=xxx (replace xxx with the full url of the game you want to play.) The difference is that the latter doesn't have the live transcript feed and only has the statistics page (note a different address than what the demo site uses). You can't single out individual players from the statistics so there's no disclaimer notifying the player of the data collection (which might unnecessarily scare off some players.)

The statistics page collects information and displays the urls of all games played this way, so don't use this for trying out something you don't want the whole world to see.

How it works

I've added AJAX calls inside Parchment's UI code at appropriate places where it sends the player's command and the game's response to the server. On the server a PHP script receives the data and saves it into the database. This is all done behind the curtain so for the player the gameplay experience is exactly the same as with unmodified Parchment.

The statistics page collects some extra information about Inform 7's internal error message and library message codes. This requires some modifications to the game file itself. The Raising the Flag on Mount Yo Momma game used as the example has been modified to include this:

That code shows the error codes and the library message ids after every turn in the prompt, for example #LIBMSG:40#>. When the modified Parchment sees these strings it saves the information in them and removes them from the output so that the player doesn't see them. Played on any other interpreter the game would of course display these, so the last line of code warns the player that they shouldn't be using this game file if they can see the control codes.

Unfortunately the error codes and library message codes aren't very reliable; some parser errors don't seem to have an error code and some library messages don't have an id. Most noticeably the "unknown verb" error doesn't have either, so it doesn't show in the statistics. I don't actually know what the codes really do, I just noticed they're there and used them; future development would include using custom codes for different situations in the game.

Future plans

• Test it more and fix bugs so that it could actually be reliably used.
• Clean up and comment the code so that it could be packaged for people to install on their own servers.
• Create a service where people could have saved transcripts and command statistics behind a password so that they could improve the game for later builds or use Parchment to save beta-testing transcripts.
• Add more information and charts to the statistics page (suggestions welcome.)
• Try to convince Parchment developers to include this functionality into the main build or provide hooks so that this could be added as a plugin. Right now I'm just modifying Parchment's code directly, so whenever Parchment is updated the files have to be edited by hand again.

What would you like to see done with this?