1st October '14 by Jaime Cross
We're on to the second proper part of my little series of blogs covering the use of audio in storytelling for games. Previously on Team Junkfish I spoke about the use of music in games and how we're going about those ideas in Monstrum. This time I'm going to talk a bit more about the use of sound effects and how they build up the world and everything in it.
Before I start, here's a Gamasutra article by Andrew High on game music that is very much worth a read.
The IEZA Framework
Let's begin with some theory first and explain some terms that I'll be using later. I'll try keep it short! The IEZA Framework can be used to describe the sonic environment in a game, and is a good starting point when conceptualising the sound of a game. It looks like this:
Basically, it breaks down game audio along two axes: “Diegetic” and “Non-Diegetic” which we've covered before (primer: diegetic when something originates within the game world, such as footsteps, non-diegetic: something outside the world, like music with no in-game source) and “Setting” and “Activity” which relate to, in basic terms, the overall environment and mood of the game and the actions that can happen in it respectively. From there it gets broken down into four quadrants:
“Interface” - Cues that are used to provide information to the player, such as health pickups or menu selections. It lies on “Non-Diegetic”, as your character won't be able to hear you changing menu settings as it's disconnected from the in-game world, and “Activity” as the player is performing an action in order for the sound to happen.
“Effect” - Relates to things like footsteps, gunshots and other sonic responses to player actions. As such it lies on the “Activity” axis, with the sounds falling on “Diegetic” axis due to their in-game world origins.
“Zone” - Relates to the sonic space that the player currently inhabits and should be designed to replicate that environment. This lies on the “Diegetic” and “Setting” axes, as it has in-game origins, but it's not being driven by player actions. Think environmental sound effects, traffic, wind, that sort of thing.
“Affect” - This one is a bit nebulous, as it relates to audio that somehow “affects” the player, such as setting a mood via music. This could be something like an orchestral sting when a monster jumps out at you, or addressing an area you happen to be in (like bagpipes in Scotland, for one stereotype). This sits on the “Non-Diegetic” and “Setting” axes.
By following this framework we can define different types of sounds, and from there we can discuss how they assist in world building and story telling. For a more in-depth read check this article by Richard van Tol and Sander Huiberts.
So now that that's out the road, let's get down to business. I'm going to try break down the audio from a few games into these categories and explain how they help with world building and storying. So without much further ado...
Spoiler: It's not that scary.
Animal Crossing: New Leaf
Probably not a game that people would expect, but "Animal Crossing: New Leaf" (I'll use Animal Crossing in reference to this title from here on it) has some rather interesting and detailed sound design choices that help build up the world and the characters in it. In the game you play as the new mayor of a town, and it's up to you to bring in new residents and drive its development forward. The game rotates through a day night cycle in real time, and also has a weather system in place, both of which have an effect on the sonic background of the game. There are a couple of points that I quite like about the game, so I'll break them down into the IEZA sub groups where applicable in this.
Starting with some of the more obvious ones: footsteps! The easiest way to give somewhere a sense place is through footsteps (and reverb). Changing the sound for each surface is an obvious thing, but it it's still rather effective. Going from town's dirt ground, down a stone ramp to the beach and then across the shoreline with the sound changing beneath your feet just feels right. Even the carpets in your house sound different depending on their texture. Similarly, when the rain drifts into town the footsteps naturally change to reflect this, but they aren't alone in this regard. If you go digging in the rain the mud takes on a denser, wet sound as you throw it about and pack in holes. It's small, it's subtle, and yet it makes the game world seem a bit more plausable, a bit more believeable, for it.
Another interesting thing in Animal Crossing is the speech. When you talk to characters in the game they start “speaking” in a seemingly garbled language which, if you listen closely, seems to replicate what's in the text box! While any sort of chatter could have done this vocalisation, in addition to giving the townsfolk some degree of variation with their “voices” (although they aren't completely unique) adds to their own character. When Timmy or Tommy Nook are trying to sell you something and dip into their little quiet sales pitch, which is reflected in the text by a smaller font as well, you can believe that they're talking to your player. This also ties into...
A further nice little touch in the game is the text entry. As you enter each letter your character seemingly reads it out for you, giving them their own voice among the crowd. It's particularly fitting when another character asks you for an answer or you send out a message and you get to vocalise what is said (to some degree).
The town itself has some obvious zone related sounds effects, namely the beach area and when you stand near the river as you hear their respective flowing waters. But it also makes use of ambient sound to alert you to collectable insects, giving certain families or bugs their own calls so that you can not only tell where an insect is but what type it may be. This is a nice little touch, as the developers could have went with one stock insect noise but instead populates the world with a variety of life. It also makes it easier to try and find insects you haven't caught for your collection too. There are also items that float into the town on balloons, which tend to duck the music to allow a wind sound to play as the balloon comes into view. It is a small way to attract the players' attention to a focal point by utilising environmental audio.
On a side note, the 3DS actually has a surprisingly decent HRTF implementation with its speakers that doesn't get enough love or attention. Nintendo really went all in on the 3D! But implementation is another key part of game audio and is often overlooked by developers. This is, however, a blog post unto itself and has a fair amount of tech considerations too! So perhaps further down the line!
It's not the Old Grey Whistle Test, but still.
There are also a few instances of diegetic music in the game. Most obviously is Club LOL, the nightclub. It occasionally features the guitar playing canine troubadour K.K. Slider, on top of general club nights which he DJs under the psuedonym DJ K.K.. On Saturdays K.K. Slider will perform various songs, which you have to actively become the audience for by taking a seat, take requests and pass copies of tracks on to you to play in your house. Similarly, when travelling to and from the island you'll also be greeted with a little sea shanty from the boat's helmsman, Kapp'n. While the music remains the same, he tends to change the lyrics quite a bit, singing along via his synthesised voice.
Lastly, the use of music in Animal Crossing in terms of world and character building is pretty intriguing too. There is a “town theme” that each potential villiager has their own variation of. This little jingle plays whenever you talk to someone, and manages to put across some aspects of their personality. If you don't like the original you can actually change the tune that's played, allowing you to impart some of your own personality back into the town too! This is limited to 16 notes and the scale of C Major, but is still a nice feature that plays on the game's customisation and town building motif. If you play the game different times of the day you'll also get different background music, reflecting how active the town is.
There's also a little throwback/Easter egg in the Nook brother's T&T Mart, which riffs off of the instrumentation and themes from the Wii Shop Channel's music. Of course, no Animal Crossing game would be complete without “Tokata's Song”, a notable musical 19-note Easter egg by Nintendo sound designer Kazumi Totaka that appears in a number of different titles which he has worked on. He's also the source of K.K. Slider's name and the voice of Yoshi. The more you know.
These are only a few examples of Animal Crossing's well thought out and crafted sound design, and only from one game too. Journey is another excellent example of using sound effects to build up the game world and story is ThatGameCompany's “Journey”. Sound designer Steve Johnson did detailed breakdown of his process which you can read here. Visually the world of “Journey” is stunning, but the sound both it and its inhabitants make really bring the game to life. The article also has sound examples, so give it a read! Similarly, the "Abe's Oddysey/Exoddus"' sound design allow players with little or no vision to navigate through and complete the game. I'll talk about that in the next major part of this series, which will have look at accessability and Audio games, and might even touch on the implementation side of things too.
Listed and Further Reading
Caleb Bridge - Creating Audio That Matters (Gamasutra) - Touches on audio immersion
Andrew High - Is Game Music All It Can Be? (Gamasutra) - Excellent, lengthy article on game music
Steve Johnson - The Sound Design of Journey (Gamasutra)
Darren Korb - Build That Wall: Creating the Audio for Bastion (GDC Vault)
Kirk Hamilton - How To Compose A Great Town Theme Song In Animal Crossing: New Leaf
Richard van Tol and Sander Huiberts - IEZA: A Framework For Game Audio (Gamasutra) - Breakdown and examples of the IEZA framework
IZEA Framework (Wikipedia)
Kristine Jørgensen - Left in the dark: Playing Computer Games with the Sound Turned off - Talks about how audio provides information and how a lack of sound changes players' perceptions. It's part of "From Pac-Man to Pop Music", a collection of studies covering a variety of different game audio topics.
Karen Collins - Playing With Sound - Another of Collins' books, this time on how players interact with game audio as opposed to listening to it, and how it can alter players' experiences.
Andy Farnell - Designing Sound - A technical book on sound design, and practically a bible for anyone interested in PureData and DSP.
David Sonnenschein - Sound Design - A more general sound design book, but covers a lot of the theory behind it very well.
Use Distinct Sound/Music Design (Game Accessibility Guidelines) - More of a segue into the next possible topic, it has examples of a blind player playing through "Abe's Oddysey" and beating Ed Boon at "Mortal Kombat" through sound alone.
9th September '14 by Andrew Bean
Today I thought I'd explain how the journal system in Monstrum is implemented. The journal is our equivalent of a quest/mission log in many other games, and has a few noteworthy features:
It is diegetic, which means it exists in the game world itself, instead of, for example, being rendered on a 2D plane in front of everything else. We chose a diegetic interface as it is generally more immersive, and also works better for the Oculus Rift.
The information in the journal is rendered on the page itself, following the curve of the paper. This makes the journal feel more natural than, say, having text float above the page.
In order to create the effect of having the UI rendered on to the page, the journal model is first split in to two - the cover and the page. This is so the two meshes can be UV mapped separately. The cover is just a standard model, but the page is UV mapped with (0,0) in one corner to (1,1) in the opposite. A separate camera is pointed at a custom UI system and draws to a RenderTexture that is displayed on the page. At this point, we can see the journal info on the page. The trouble is, we can't interact with it!
To solve this second problem, we have to start with the player. The player starts an interaction with any given object by looking at it. Therefore, we raycast along the camera forward, looking for the collision mesh of the journal page. If we miss, the player isn't looking at it. If we hit, how do we know where on the page they are looking? Well, Unity provides the texture coordinates of a hit from a raycast. These will be mapped between (0,0) and (1,1), as defined earlier by the UV map. All we need to do now is use the Camera.ViewportPointToRay function, and cast the returned ray, masking for our journal UI elements. At this point we can do whatever we like with the returned hit data (button clicks, mouseovers, positioning the journal cursor). Hooray!
I hope this technique may be of use to someone!
5th September '14 by Adam Dart
Once again, we've been working on some super cool stuff that we would rather keep secret for now. However, I thought I would share some interesting videos that may, or may not be related in some way or form. (Maybe even set the mood by watching the first 3 whilst listening to this playlist of the Alien soundtrack... )
1 – Macropinna Microstoma. A cool semi-transparent fish.
2 – Squid versus Fish
3 – Life in the Deep Sea
And one last one...
Take from this what you will.....
Adam and the Arts.
3rd September '14 by Team Junkfish
Hey, Grant here, making another ham-fisted attempt at arranging words to form coherent sentences.
Procedural games are a tricky business. As I've mentioned before, when you aren't in strict control of things like the level layout and scripted events it adds a whole host of design considerations. One of the most important of these is how you actually teach the player to play the game, and this blog will discuss some approaches to this problem. But first, a little history.
In days of yore
In the early days, back when the market was young and game design wasn't well understood, many games took the standard software approach to informing users which was just to supply a manual with the product. The player was expected to read this to learn the controls, systems, and sometimes backstory of the game.
With the rise of consoles and the growing complexity of games, tutorials have become standard. Manuals still exist but they're really just tokens now, little books that add weight to the box likely included because many gamers (myself included) have a sentimental attachment to them. In fact with the popularity of digital distribution the idea of a modern game without a tutorial is unthinkable.
The problem with procedural games
Most games implement tutorials mainly as an early part of the game, usually an easy section with no threat of failure and often worked in to the story (e.g. boot camp/training for military games, escaping jail for stealth games, etc). Tutorials are usually as long as a game needs them to be ranging anywhere from a few minutes to several hours. Sometimes they are spread out through the entire game, explaining new mechanics as they are introduced.
This formula doesn't really work with procedural games, which are made to be played multiple times. Having to play through the tutorial section over and over again would quickly get tedious, especially if it takes a long time to complete. The general idea is that players should be able to jump in and play once they know what to do.
How do other games solve this?
Different games get around the tutorial problem in different ways. Games like Daylight, Binding of Isaac and Eldritch give the player all the controls right at the start then turn them loose, gradually introducing item and enemy mechanics through gameplay. This means that new players can read the instructions and take their time getting used to the controls, whereas experienced players can just get on with it.
The first room of the game, plenty of time to memorise and play about without any threats
Other procedural games, such as FTL and Spelunky, take a different approach. Since they have more complex controls/mechanics they run with a more conventional system and have a short tutorial section. How they avoid repetition problem is by separating the tutorial section from the main game and making it optional. Interestingly enough this approach isn't actually unique to procedural games, for example Deus Ex had this system back in 2000 which shows the developers expected people would play it more than once. (And by god they were right, the game even spawned a meme to that effect).
How are we doing the Monstrum tutorial?
Honestly? Right now we're not 100% sure, but we do have some plans and ideas. We are in an unfortunate situation where neither of the aforementioned solutions alone will really work for us. The things we want to teach the player to start off with are a little too complex for the first, but not really extensive enough to require the second. Our first approach was a combination of the two: a small section contained entirely within the first room that new players would play through to learn controls and a few necessary mechanics while experienced players could breeze through in around 10 seconds and then get on with the game. However we found that even at such a short length, this became frustrating to play through each and every time the game was run. In response we then stripped this section down to a much more basic form, but this resulted in us not being able to teach the player as much as we wanted to, so currently we are revising our approach.
What we're aiming for now is a more spread out tutorial system, teaching the player as they go through the game. Ideally how this should work is that when the player encounters a new mechanic, they should already have the prerequisite knowledge on how to interact with it, with just a little help from us to guide them. To do this we plan on using:
The player has a journal that tracks their current objective and gives them hints about what to do next, which combined with what they already know will allow them to complete it. E.g. They find a fuse box and gain the objective of finding a fuse and restoring the power. By the time they have found the fuse and returned to the box they should already have learned how to pick up and use items (Prompts) and that they can interact with the fusebox (UI effects)
Strategic Item placement
A difficult one to explain so I'll just give an example. At a later stage in the game the player will be required to grab and move a large environment object, however before they can do this they will need to reach a switch somewhere else in the room. What we could do is block the path to the switch with the movable item, so that by the time they actually need to move it they should already know how to do so.
We are also considering the idea of providing a setting to toggle parts of the tutorial (Such as prompts) on or off in the options menu. That's all I really have to say for now, but if you're at EGX at the end of September you can see these systems for yourself! Because we will be there! With the game! Subtle hints!
Avg Coherency Rating: 46%,
Gra-WAIT. HOLD UP A SECOND.
Wait a minute right here. I wrote a blog without sharing stuff I think is cool! This will not stand, I must post something, but what should I choose? My favourite tutorial sections? Sure here you go:
Call of Duty 4: Modern Warfare
What I like about this is that it records the time taken on the course at the end and recommends a difficulty based on your performance. That's neat! That's a neat thing!
Star Wars: The Force Unleashed
Holy shit! Did you see that bit with the gate? In addition to making you feel like an absolute badass while you learn the controls, the game is also teasing the kind of world-destroying power that you will have by the end of the game.
Also check out Egoraptor's Sequelitis episode on Megaman X for an example of a fantastic tutorial. (Language warning though. Language. Warning.)
There we go. Now I can sign off properly.
Avg comma to word ratio: 7:1.
29th August '14 by Jaime Cross
This is one follow up to my previous blog which discussed storytelling in games. You can give it a read here. My original plan for the second part was to speak about the use of sound effects in story telling and world building, however I thought it'd be an idea to talk about how we're actually going about the music side of things ourselves!
In Monstrum we are aiming to use both diegetic and non-diegetic music for a few different purposes. The most obvious use of non-diegetic music are the monsters' themes, so let's start there. Simplifying some statements down a bit, these are used to provide information (i.e.: the monster is chasing you), context (which monster it actually is) and emotional content (trying to evoke a certain feeling). I've spoken about the “hows” of the Brute's theme before in detail here, including of the sound design that's used, so I'll give you a little summary of the decision making, the “whys”, that went behind it and what I was hoping to achieve.
First, let's talk about the Brute's musical flow:
Here we can point out how the musical themes provide information and context. There is the initial Wandering theme, which is used as players walk around the ship before being spotted by the monster.
The information players can learn from this is that they are "safe", but it is actually quite interesting due to the lack of context it provides. In a game that is going to be different every time you play, it doesn't make sense to give everything away from the beginning. It seems obvious, but it ties into how the Stalked theme is used and how it replaces the Wandering theme. You now know what else is on there with you, so the original gap in your knowledge has been filled, but you are also not actively being chased or hiding, which is when the Wandering theme usually plays. So while the information remains the same, the context is different, and this new emotional content is represented musically.
Brute Stalking theme
Each monster also has its own Stalked theme, which utilises some of the thematic and instrumental motives that are established in the Chasing music .
Brute Chasing theme
The Brute has an almost constant, unflinching rhythm during his Chasing theme, provided by the pounding drums, cymbals and background pulse. This is to reinforce his nature as a physical, charging force that has a singular focus and goal. On top of this are the metallic, almost mechanical screams and squeals that sound like metal buckling under strain. This was mainly an emotional content decision. I wanted the listener to feel uncomfortable slightly stressed while being chased because, well, it would be. I also wanted to somehow represent the ship in the music, so this was a decent compromise too as it also plays into some of the randomised creaks, thumps and squeaks that happen as you explore. These are the elements I bring back into the Stalked theme, because trying to sneak around while drums thunder in the background is somewhat jarring...
Anyway, time for something a little newer:
We're been working on the Hunter, so here's a little snippet from one of its themes. I'll do a more detailed break down of the hows and whys behind it later on though, because
Jonesy The Guardian watches for any and all leaks.
Now a little about our use of diegetic music, and how we're using it to build up the game world. Diegetic audio is something that has a source on screen or in the game (or film) environment. Basically the characters on screen would be able to hear it. I've briefly spoken about the radios in a previous blog post, and their implementation has evolved a bit further since then.
The 70's was an interesting time for music, with lots of genres and sub-genres coming into the mainstream. There are a few common musical tropes that hang over from that era, such as funk, disco and punk, so if we're going for general world building and theme setting it makes sense to focus on the easily recognisable ones. Canterbury Prog would probably be lost on a more than a few people. So how are we going about integrating these ideas into the game, and why do they have to be diegetic in the first place?
Also heard in our trailer...
From a gameplay perspective we are using them as part of the distraction mechanic. In Monstrum each monster will react to sounds that occur in the game world. For example, if you run around and throw things about a lot then the monster's more likely to hear you (and thus chase after you) compared to you walking around quietly.
Here's where the radios come in. Being a portable usable item they can be thrown into rooms, corridors, whatever you like, while active and will act as an audio source in the game world. As the music is coming from a specific object, we have also tweaked the song so that it sounds like it's coming from a fairly bad mono radio, like the character and monster would hear it, as opposed to the general themes that are “just there” in the background. This makes logical sense for how the item is used, and also proves a sense of place for the player and their interactions.
For an extra trick the “signal” deteriorates as you go further into the ship's underbelly. This has a few benefits, one being that it serves as a rudimentary compass, but it also helps keep the tone of the darker, grittier areas intact, as a funk track in the middle of the cargo hold maze might dull the atmosphere a tad. While these would be perfectly functional just emitting noise, the opportunity to do something a bit more interesting on the whole was too good to turn down.
And that's why these films will have soundtracks
We're hoping to have the record players with different (and possibly even swappable) LPs working at some point soon too, although these will be stationary sources.
Hopefully this'll give you a bit of insight as to how we're doing things and following our own advice :).