LECTURE WEEK 10 MMST 12016: Audio at Play

Audio and computer games.

Introduction

In this weeks lecture we look at digital audio and video games. The video game industry is one of the largest contemporary cultural industries, and for that reason it is important to spend some time considering some of the issues particular to the design an development of audio for games. Consideration of audio in games is also a good point from which to think about the role of sound in creating a "virtual reality". I use inverted commas because, in most games, we are not talking about a virtual reality environment as the term is usually defined. While current game design aims to provide an immersive experience for gamers, the visual elements of games remain two dimensional, ad fixed on a screen in front of the viewer. However, games are becoming increasingly immersive and increasingly realistic, and it is possible to argue that they represent the first popular application of virtual reality in popular culture. With the advent of inexpensive surround-sound-capable audio cards and surround speakers, the audio component of games leads the way in this regard.

This week's material incorporates a "guest lecture" by gamer and Multimedia tutor Peter St James. Peter's lecture illustrates some of the many way sound contributes to the experience of game play, and emphasizes the subjective experience of the gamer.

Computer games, contemporary culture and audio: an historical perspective.

In 1972 the US company Atari released the simple tennis game Pong. On the left and right hand sides of the screen was a white stripe, representing a paddle. A white block representing the ball flitted from side to side. Game play involved manipulating your paddle and keeping the ball in play for as long as possible. The game took its name from the real-world game that it resembled, and from the sound the game made when hit by the panel - Pong (Herman et al). Six years later, the release of Taito's Space Invadersushered in the golden age of arcade games. A year later Atari released the first successful home video game console, the Atari 2600. In the late seventies and particular in the 1980s, arcade gaming and its home equivalent became important aspects of mainstream youth culture. Gaming on home computers generally lagged behind the arcade and the console, but caught up in the 1990s.

Throughout this period, the use of sound was critical to the success of these games and integral to the gaming experience. One of the more popular early games was Berzerk, released by Stern in 1980. Berzerk represented a breakthrough in digital audio production as it marked the first time in gaming that digitally synthesized voice was used. Berzerk made use of several phrases, including "coins detected in pocket" which would occasionally emanate from the machine when it was not being played. At the time, the use of such digitized voice represented a significant investment by Stern. I have been unable to locate exact figures, but by most accounts the budget for this part of the game alone was several tens of thousands of dollars, a very large sum at the time, particularly as the game contains a total of only about thirty words.

Below is a screen shot of the game, and a link to an mp3 file containing sound captured from the game play, including some of the famous digitized voice.

berzerk.mp3

In terms of technological sophistication, and hence quality of game play, arcade games had a head start over the home console and the home computer. By the mid to late eighties technological advances, combined with the extra resources that accompanied the growing size of the console market significantly narrowed the gap. By the late eighties, the home consoles were no longer in the shadow of the arcade machines, and many of the most popular console games were never saw release as arcade games.

The rapid development of gaming technology makes the arcade/console/computer game a very interesting cultural site. While, even during the golden age of the 80s and early 90s, games became passé shortly after their release, the better ones were remembered with a fondness bearing on nostalgia. The rapidity of the development meant that games passed into the realm of nostalgia more quickly than most other cultural products. Consider another classic game, Pac Man, from 1980. Again, a screen shot and some sounds have been provided.

pacman.mp3

Pac Man is one of the few games that became known outside of youth culture (Space Invaders is another). Of course, games from the later period of the 1990s became well known as they were often part of cross-media franchises, and as such crossed over into another branch of pop culture. The recent Tomb Raider and Final Fantasy movies are good examples of this phenomenon

By the late 1980s, arcade games were no longer novel, and were being challenged by the home console market. Yet the "classic" games had come to signify something to those who had played them as teenagers or younger children. There was a brief phase where older games were revived, often in a self-consciously parodic or excessive manner. A postmodern move on accelerated nostalgia was taking place, and audio was a large part of the formula. Have a look at the next screen shot, from 1987's "PacMania". Also listen to the audio file that has been supplied.

pacmania.mp3

You will notice that the video component of the game was much more sophisticated. The game played on a screen that scrolled horizontally and vertically, and much of the screen real estate was taken up with images of homage to the original Pac Man. The audio component too had its similarities, but the excessive use of stereo panning and ostentatious use of complex music and sound effects contrasted sharply to the quotations from the original lo-fi soundtrack. The high-resolution of the video component did not add to game play, in fact it detracted due to the scrolling it necessitated. Yet the overall game design made a point about the place of Pac Man in the history of the arcade game. I suggest that the elaborate audio component needs to be seen in the same fashion.

This self-consciously nostalgic use of gaming's back catalogue is best exemplified by the phenomenon of "retrogaming", where the arcade games of the retrogamer's youth have become significant objects of nostalgic recollection and are preserved, revived and celebrated. In large part this phenomenon, which is an aspect of a larger "retro" sensibility in contemporary popular culture, is a product of digital technology. Fabio Da Silva and Jim Faught make an interesting link between this fashion and the technology through which it is practiced and disseminated:

The recent preoccupation with nostalgia has been constructed on the selective isolation and petrification of cultural objects who's existence was predicated on another material basis of production. By transporting artifacts, unencumbered by the negativity of the past, they may now be used to fictionalize past and present experience. As presently constituted, nostalgia requires a collective emotional reaction toward, if not an identification with a symbolization of the past. (da Silva and Faught, 48-9)

Da Silva and Faught are talking about technology that allows the use of one form of "petrified" media in another, whereas the PacMan/PacMania example above is an example of one cultural form cannibalizing its past. Their point remains relevant. Retrogaming however is a prime example of technological petrification; through the use of emulator software that runs on standard PCs, the arcade games that originally ran on idiosyncratic, custom hardware have become artifacts. The games live on,albeit serving a new purpose, long after the demise of the original hardware and the lack of a market for outmoded games. If you want to know more about this interesting subculture, go to a search engine and carry out a web search for "retrogaming".

As the children and teenagers who played these arcade and console games grew up and took their place in adult society, a significant proportion of them retained a taste for gaming. While for some this remained a nostalgic interest, the majority have retained a current interest. Microsoft's current attempt to enter into (or some would say dominate) this market attests to the degree to which gaming has become a major and mainstream cultural industry. The industry is now very large, and there are multimedia firms that specialize in the production of audio for video games. One such firm is Sound Tomatoes, and their web site is interesting.

Before moving on to other issues of audio and gaming, I will leave you with one more link. GameSpot has an online feature on the history of computer game sound that is worth investigating for a more thorough history of video games, and video game sounds.

Games, Audio, and Virtual Reality

The guest lecture below details one gamer's experience of sound in an online game world. The online world of Anarchy Online does not fully accord with the traditional definition of virtual reality: The character is represented as an avatar on the computer screen, rather than the gamer entering into the world and navigating through it wearing virtual reality goggles. In other words, the gamer is never "present" in the Anarchy world. Yet a well-designed game built around a virtual, navigable space can immerse the gamer through game play. As Peter St James suggests, audio is a very large part in this process.

Before you read Peter's account I would like to direct you to two readings from Heim's book, Virtual Realism. The first excerpt (pp 42-50) discusses approaches to what he calls virtual realism. Bearing in mind that Heim is, in fact here, reviewing an exhibition of virtual reality art, much of the discussion situates Heim's view of VR in relation to two opposing camps: particular cyborg theorists who suggest that the augmentation of our experience with (computer) technology is inherently and unproblematically liberating, and neoluddites who see little of value in VR. You do not need to concern yourself with these contexts, but should focus on Heim's tentative definitions of virtual realism. As you read the section on Anarchy Online. you should consider how well you think Heim's approaches apply to Anarchy Online. Remembering back to the discussion about the nature of the visual and the aural in Lecture 1, you should think about the place of audio in virtual reality.

The second excerpt (pp 94-98) discuss an example of "world construction" using audio. You should consider the ways in which audio can be used to construct an environment as you read the following section. You should also reflect on how such "world construction" might aply outside of gaming and VR, and consider the qualities of sound that make it an appropriate medium for such an enterprise.

Readings: Heim, Michael. Virtual Realism, Oxford University Press, 1998.

The role of audio in developing gaming environments

Peter St James

As the world of computer gaming comes closer to reproducing real life environments, the role auditory stimulus plays becomes more important. The human body understands its world through five senses. The computer interface immerses the human into their world through three of those senses. The most dominant is visual. Various peripherals explore the sense of touch, predominantly focusing on vibration. How then does the auditory sense figure in the reproduction of real world interaction?

"Auditory feedback can present further information when the bandwidth of graphic information has been exhausted, as is often the case with the present emphasis on graphic presentation." (Lombardi)

Bandwidth here refers to the amount of information the eye can register through visual stimulus. The difficulty facing game developers is not the limitations of hardware in developing graphical representation, but the limitation of the human eye in its ability to absorb the amount of environmental information presented visually. Here in lies the importance of sound to the development of gaming environments. As Lombardi puts it: "[b]y expanding conventional interfaces in another dimension, sounds make tasks easier and more productive."

In many areas sound is more suited than graphics for delivering information. Game developers are now understanding that sound can also free up graphic areas for other uses. The solution to improving graphical bandwidth issues was to replace many graphic features with auditory ones. This increasing dependence on audio to produce stimulation has helped the computer interface approach replication of the real world.

The computer chip however has a long way to go before reproducing the unpredictable nature of human interaction. So it is through networked or online gaming environments that the combination of real human interaction with computer generated 'worlds' produces the closest thing to 'reality' available to the gaming enthusiast. With this in mind let's take a look at one of the leading online gaming communities Anarchy Online.

Anarchy Online (AO) is a game set in a futuristic time frame on a planet called Rubi-ka. In considering the role of audio in this online gaming environment we will look at audio as ambience, navigation and narrative. Before we delve into the interface lets have a quick look at the video introduction to the game. First listen the the audio file. What imagery is formed when listening to this file? What parts of the auditory affect the images you visualise? On the resource CD there is a video file - intro.exe - of this same introduction. You should view this if you are able. Is it what you expected?

NAVIGATION

Navigation describes how a gamers use the interface that allows them to interact with the gaming 'world'. A veteran gamer will subconsciously respond to generic sound effects accompanying most contemporary game interfaces. The common 'bing' is universally associated by gamers with a confirmation of a command entered into the interface. Just as recognisable is the 'dud' sound letting the player know that the interface command they issued is not correct or available.

The AO interface provides a number of sound effects supporting keyboard and mouse commands. These include a button press, open, close, and drop.

Environmental support of the interface exists through The physical environment of AO's interface is supported by sound effects such as a door opening and closing, that confirm a command issued with a graphic game element.

The AO game sounds are saved on my computer's hard drive where as the graphic components are controlled through the game's online server. As I only have a 56k modem, some of the graphic responses to interface commands can suffer lag. More times than not it is the audio response that lets me know the game has received an interface command.

"As each type of switch in real-life has its own texture and resistance, each type of software switch (e.g. an on-screen button) could have its own sound which the user would associate with that button and that function. As familiarity with the sound grows, the user will associate a sound not only with its respective button, but also with its function. So not only will the user know by the auditory cue their action was registered, they will also know, with utter immediacy, what action was performed - without having to read a label or other description within a dialog box." (Lombardi)

AMBIENCE

The following are examples of background music used in AO. Clan, Omni, Forest, Cave. They do set the mood and compliment the atmosphere but when things are moving along I hardly notice the music anyway. When I asked some other gamers about the AO background music a couple responded that they listen to mp3 files while they play. Most agreed that they turn the music off as well.

Music has a tendency to blur into the subconscious when I concentrate on something. Music in the car can become a background noise if I focus on alternate things and the same happens with computer games. When I become focused on the game narrative, the background music ceases to exist. It is here that background music developed to enhance the ambience of a game can play on the subconscious supporting the 'feel' of the environment.

One of the first things that I noticed in playing AO was the inclusion of various weather elements in the rendering of the environment. During a session of playing I can watch the sky cloud over and hear the thunder rumble in the distance. Very soon there is lightning and it begins to rain. Later I can see the sun come out and the hear the birds begin to sing. These subtle effects reflect real-life experiences and their gradual development within the environment help support the ambiance of 'living' in a simulated world. I recall having to take cover once because the 'storm' was so bad I couldn't see. The rain was belting down with a howling wind and thunder and lightening crashing down around me. It was the most stimulating audio environment I have ever experienced, created solely for the purpose of ambiance.

AO has visually stunning graphics and the aesthetics of the landscapes is highly developed. However, as you can see for yourself below, if you only engage the landscape visually all it is is a picture. Admittedly, in-game I can move around these landscapes as if they are three dimensional, but they do not become an environment without the auditory component. To create the ambiance of a landscape, the gaming environment relies on sound to 'solidify' the feel of area around the player character. Recently I logged on to play AO and after a short time was puzzled because something wasn't quite right. I realised I had forgotten to turn my speakers on. Subconsciously I had noticed something askew. The environment I was playing in seemed 'flat' because it was missing the audio integration. The realism of environments in AO is not improved by sound, it relies on sound. By including a variety of sounds that you would normally hear in real surroundings, the player feels more 'at home'. This for me makes the gaming surrounds seem more realistic and easier to 'slide' into. To me the planet of Rubi-ka exists. If may be artificial but it exists as a real place regardless if that space is electronic and intangible. When I play I exist in that space and the sounds are central to the accomplishment of this immersion.

Examples of AO landscape sounds

Cityscape           Forest           Ocean Plains          Nightclub          Cave

Another interesting feature of AO is the inclusion of footsteps. Taken for granted as a real environmental auditory event, they are adopted in-game quite easily to signify movement and also surface. Footstep - concrete, sand, and water. The sound of npc and player footsteps around me also adds to the realism of the game. I can stop at a closed door and hear someone walking around in the next room.

I categorise non player characters (npc) into three subclasses of automated, animal and humanoid. To me the audio that supports playing with these npc are crucial to the success of a game. Poorly developed npc sound effects would be enough for me to 'trash' a game. I also hate those games that have the same sound effect repeated over and over. Like the music it becomes a background sound that needs muting. Npc sound effects need to be loud and intrusive so that the effect of interacting with npc has an atmosphere, whether it be fighting, searching, controlling or negotiating. AO draws its npc sound effects from a bank of sounds for each different automated, animal or humanoid npc. The game moves through the sounds randomly responding to what is happening in the game environment so that no sequential sound is the same. You can shoot an npc a number of times and get a different 'npc hit' sound each time. This makes AO an excellent game in terms of npc interaction. Having such a large variety of sound responses not only increases the enjoyability of interacting with computer controlled graphics, but saves them from an existence as mere gaming obstacles (a simple analogy to explain obstacle would be the ghosts in Pac Man).

Automated npc sound1, sound2

Animal npc sound1, sound2, sound3, sound4, sound5

Humanoid npc hit, dying

As with most contemporary games AO provides control over the audio elements of the game. You are able to switch music and sound effects off and on and set the levels. AO also includes the option to provide 'battle' music where chosen. The player can have battle music only appear when their opponent is tougher than they are, or set it for all battles. In this way the music provides an indicator free of graphic stimulus.

Background music when fighting opponent - tough, medium, slight.

As with most contemporary games, AO provides an interface 'slider' to adjust audio levels. When gaming I like the sound effects to intrude upon any background sounds. In this way the audio is my main indicator of event changes or response(s) to actions.

NARRATIVE

Most narratives in AO require interaction with npc. A common narrative is a dungeon 'shoot em up' environment. When moving about inside a building, cave system or even a forest a player is surrounded by sounds. In amongst those sounds is the sounds your opponents make.

"Our ear is able to define the direction on a sound and the distance to its source more or less precisely." (Simanenkov Dmitry, http://www.geocities.com/guitar_effects1/3d_sound_new/3ds1.htm).

AO uses pitch and attack to create distance for its npc sounds. A lower frequency pitch combined with a longer attack signifies distance. As the npc gets closer the frequency becomes higher and the attack is shortened. In this way by listening to the sounds I can predict how far away the npc is. The sound also becomes clearer when the character faces the direction of the npc, so by turning on the spot I can direction myself on the npc Most of my direction sense in-game is done by sound.

I wasn't too sure whether npc voice responses accompanying elements of the game would be navigation or narrative. My choice came down to the fact that the voice responses were not a consistent g.u.i. (hraphical user interface) support but rather an element of the game. Voice responses are not used extensively in AO. They only exist in the character creation area(s) and the 'terminals' used by characters to create narrative, maintain their characters, and store items. These areas would be the initial environments used by new players so the voice responses are auditory support for learning how to use key components of the game.

Anarchy online enables the gamer control (no matter how artificial) over narrative paths taken by the gamer's character. It's a bit like reading a book but better. One of the characters is mine and I play a part in writing the story. Its the reason I play. Like a good book, if its a good story then I keep on reading. So the narrative has to be good for me to keep playing a game.

To me the narrative is successful if all of the sounds are working together to improve the reading of the story and make the story-line come alive. So the better the audio support of navigation and ambiance, the better the response of the player to the game narrative. If the audio helps the navigation of the interface, game play is smoother and more acceptable to me who is busy typing, pressing, clicking an moving peripherals attached to the computer. If the background sounds make the surrounds come to life then I can immerse myself in the game. It becomes more 'real' to me and makes the game narrative more enjoyable. And if the sound effects improve the quality of my interaction with the environment, other players and npc then events are more stimulating. This stimulation is the key to the life-span of a game. When looking at AO we are talking about a game that will be played over years, not days or weeks.

There is a little saying "Life is a book." Gaming environments like AO create a life of their own. The believability of these environments depends on how well sound is used to develop them. I can login to AO, close my eyes and experience the world through what I hear.

WORKS CITED

Fabio B. DaSilva and Jim Faught, "Nostalgia: A Sphere and Process of Contemporary Ideology," Qualitative Sociology, 5 (1982), 47-59.

Leonard Herman, Jer Horwitz, Steve Kent, and Skyler Miller. "The History of Video Games", Gamespot, http://gamespot.com/gamespot/features/video/hov/, n.d., accessed 2002.

Lombardi, Victor, ""Audio on the Internet", Noise Beteen Stations, http://www.noisebetweenstations.com/personal/essays/audio_on_the_internet/, 1997.

Glenn MacDonald, "A Brief Timeline of Video Game Music", GameSpot, http://gamespot.com/gamespot/features/video/vg_music/, n.d, accessed 2002.

Siamnenkov, Dimitry, "3D SOUND: History, Theory, Hardware and Software", http://www.geocities.com/guitar_effects1/3d_sound_new/3ds1.htm, 2000