For Science! For real!

If you are reading this, you are probably from reddit or the Escapist. I am pretty sure, since this blog does not get much monthly hits and the hits I do get are usually image searches for a philips screw driver. Let me say welcome and thank you. Thank you for helping me in my research on educational games, or more specifically: "How to teach software design in the form of an educational game."

Arthur, the mascot

So, how did it go? Well, I was just minding my own business, trying to write my thesis and then suddenly I have to present my research in Switzerland and I win an award in Austria for best gamification paper. I'll admit it sounds better than it actually is. Switzerland was just a poster presentation in the hallway at the end of the conference and the paper for Austria, although it mainly described my own research, was mostly written by one of my superiors. Nonetheless, I am proud of the achievement. Thank you for making it all possible.

If you know what I am thanking you for, you've only seen the tip of the iceberg. If you don't know what in blazes name I am writing about, let me explain. A couple of months ago I released a freeware game, called "the Art of Software Design". With it came 2 surveys, one to be taken before the game and one for after playing. This was part of the research I needed for my paper, but that is not the only thing I did. For a full research you need some preliminary reading (I look at A), a hypothesis (I think A gives B), you need to test the hypothesis (I'll do A), the test will give data (Ooooh, I now have C) and from that data you can draw conclusions (I say C is B, so I am right).

I am going to tell what all the other things I did for my research were . I am going to try to put it in very simple terms. If you are not interested, stop reading, accept my thanks for your cooperation and go read something else. If you want to see the full research, software and extra papers, go here. If you are still interested in this blog post you can, ofcourse, just read on. Ready? OK, go!

What did I first know?
My teacher wanted me to make a game that teaches software design, because software design has a big dullness stigma. A game might spark some interest in people who don't know what they want. I wanted to make the game fun. To do that, as a scientist, I had to find out what 'fun' actually is. The few people who have read my blog might remember my post about 'addiction does not equal fun.' Back then I ranted about the fact that game developers make a game as if they make a maze for rats. Instead of giving the player the possibility to do their thing, they want the player to do everything the way the designer intended. I said once before that games like NFS most wanted are rigged. The best part of research is that you can really dig out a subject you personally want to know about, even if it is not completely in sync with the demanded research. Just like an ice-cream man can serve himself some ice-cream, I served myself some information.

I'm learniiiing!

When making a game, most developers only look at the addiction part, but not at the learning part. When you play a game, you learn the controls, mechanics and features and you want to get better in those. People want to create their own style when playing. Even if the information is not useful, they still learn from a game. My problem starts when what you do in the game does not matter anymore. A lot of games make you lose on purpose and on that moment you are not learning anymore, you are just pushing buttons because a screen tells you to. If you ever wondered why quick-time-events are so annoying, this is the explanation. Pushing buttons when told does not make you learn, you don't practice a style and therefore you are not having fun.

My game was to be fun and educational. So I wanted my players not to become rats in mazes, I wanted them to learn and create their own style with software design. If you start researching the essence of learning (or just read this link) you'll find that in order to learn, the student/player has to have some kind of emotional connection with the thing he or she learns. In other words: if the teacher is entertaining, the kids pay attention. This is why an educational game, when done right, can teach a lot to the player. This is also why learning things, when done right, can also be fun. Education is not necessarily dull at its core. People want to do and learn things, just for the fun.

Long story short, I condensed the essence of fun and learning into 3 points that need to be balanced when making a (educational) game. They are:

• Tolerance; Don't punish the player too hard, but don't be lenient. Too little tolerance (or too much punishment) makes the player want to stop playing the game. Too much tolerance and the player does whatever they want and nothing is learned.
• Variation; Don't give too much similar situations to the player, but don't give them something completely new every time either. Too little variation makes the game dull, too much variation and the player does not understand what the game (or learning subject) is about.
• Deviation; When the player gets stuck, they need to be able to go to a different problem, solve that and get back to the earlier problem to solve it with the new knowledge they got. With too little deviation, they get stuck, don't play on and don't see the rest of the game. With too much deviation they will avoid parts of the game, because they don't feel like doing that one difficult part.

With this information I knew how learning could be fun, but I did not know enough about software design to actually teach it. I dived into a big pile of books, picked some subjects that would be possible to implement and build some mechanics around them. If you want to know what the subjects are and what they do, go play my game already. It is way easier and faster for you to play the game than it is for me to write it all down here and make you understand.


How did I do the science?
With all information and expectations in place, it was time to make the game and test it on people. This is easier said than done. Making a software design model is easy to translate to an environment. Some concepts are easy to translate, like coupling, some concepts are challenging to put into a game, like data flow, and some are so vague I needed 3 different takes on the subject before I got the mechanic right (cohesion). If you don't know what I'm rambling about, again, play the game.

Eventually I was able to make a game. This was not the version you got to play. I first tested it on some people in so called 'speak out loud' tests. These tests are as weird as they sound, people played the game and had to actively say what they were thinking whilst they played the game. It's weird, but very effective. I was able to fix a lot of bugs and counter-intuitiveness from the game.

After I fixed the game, it was ready for global release... or at least on reddit and the escapist. I made a pre-game and a post-game survey which players had to fill in before and after playing the game respectively. In the surveys I would ask questions about the subjects and the idea was that after playing the game people would know more about the subjects after playing the game.

I just forgot about one factor: the average attention span of a person on the internet. Barely one out of five people who did the pre-game survey pushed through to the post-game survey. It was a set-back, but luckily I could still use the data and I already had the speak-out-load results. It was time to make something out of my findings.

And then?
The whole thing with science is that you can not know what the results will be beforehand. If you did, you probably tweaked your results and you would be a lousy scientist. This is why it is so hard to get research funding and why you should never trust research funded by companies when it says that those companies are right on some subject.

For Science! Muhahaha

As with all research, also with mine things went wrong. There were still some nasty but rare bugs that broke the game on some points. This might explain why a lot of people did not do the post-game survey and not everyone could recall all subjects after playing that game. In hindsight, I should have implemented some netcode in which I traced people's actions.

Then again, netcode would have taken a lot of time to make and, as with all things in life, my paper had a deadline. The game was already hard to make as it was intended. I wanted multiple solutions per puzzle (variation) and a nonlinear way of going through puzzles (deviation). This meant i needed to put a lot of time creating and testing every puzzle and also that I needed a lot of puzzles. Let's just say I did not have a lot of time to do anything else besides level creating when I should be doing sciency stuff.

How did I smugly say I was right all along?
With all things considered, the research was a success. Why? People DID learn software design. At the speak out loud tests people started talking in software design terms within a couple of puzzles. When looking at the surveys the more advanced subjects were not always learned correctly, but that was because they were not practiced enough within the game. Most people were positive and we were able to reach out to people interested in software design and computer sciences, which was our main goal.

The best data I got from this research is signs that the puzzle structure was right when looking at the point of deviation from earlier. At the speak out loud tests people got stuck on certain puzzles, solved another one and returned to the previous puzzle to solve it. Note that I only saw signs, but can not claim that it really works, because for that I need to speak-out-loud-test much more people.

My research was presented at 2 software design conferences. One presentation was done by me, the other by one of my teachers and he got the award. In the end the research was a success thanks to all you fine internet people. Thank you for playing my game and I hope you enjoyed learning about software design.

Also, keep an eye on the art of software design website. There will soon be an expansion of 3 big puzzles and I will also publish the source code.