I’m revisiting this essay because I am trying to get together a book proposal tentatively titled Designing Narratives for VR: philosophical, theoretical and practical considerations. It is inspired by my teaching classes in this and seeing enthusiastic student response, and from my wide-ranging reading on related topics from dance film to Heidegger’s standing reserve. Based on my experience, no one approaches VR this way. That is neither good nor bad, but, VR has rapidly been coopted by corporate/commercial concerns (and, god knows they aren’t interested in Marinetti’s Manifesto of Tactilism… ).  I think this book would help open up important conversations. I would love to know what your favorite books on this topic are–so I can actually confirm my suspicions.   Also, is this a book you would want to read? Recommendations for publishing houses also welcome. I love University of Minnesota offerings, but open to anything.  Thanks!

 

Really pleased that my paper will be published in Transatlantica Journal, a biannual publication of the French Association for American Studies. This is based on a talk I gave in Paris last March.  This is useable under Creative Commons,  attribution, please. #VR #cinema #apocalypse #epistemology #narrative

The word “apocalypse” comes from the Greek apokaluptein meaning to uncover or reveal. In keeping with the word’s origins, I approach the concept of apocalypse not as the destruction of the material world, per se, but, epistemologically, as the end of the world as we have known it. If not the primary agent of this epistemic cataclysm, computer technology, which increasingly mediates human perception of reality, certainly reflects these changes. As might be expected, these technologies have also dramatically changed how we communicate about the world. In his essay, “Database as a Symbolic Form, ” new media theorist Lev Manovich describes this as a move from “narrative” to “database” storytelling.

As a cultural form, database represents the world as a list of items and it refuses to order this list. In contrast, a narrative creates a cause-and-effect trajectory of seemingly unordered items (events). Therefore, database and narrative are natural enemies. Competing for the same territory of human culture, each claims an exclusive right to make meaning out of the world (Manovich, 1999 “Database as a Symbolic Form”).

Manovich concludes his essay by praising the Soviet film director, Dziga Vertov whose Man With a Movie Camera, he argues, successfully creates a new kind of narrative by ordering a database of images around the “kino-eye” of the filmmaker.

This process of discovery is film’s main narrative and it is told through a catalog of discoveries being made. Thus, in the hands of Vertov, a database, this normally static and “objective” form, becomes dynamic and subjective. More importantly, Vertov is able to achieve something which new media designers still have to learn — how to merge database and narrative merge into a new form (Manovich, 1999, “Database as a Symbolic Form”).

Whereas, even today, both the novel and cinema remain mostly on the side of conventional narrative, many new media forms resist the linear arrangement of information. Of these, virtual reality (VR) most directly and completely mediates visual sense perception. In the VR headset, the world disappears, replaced by an alternative reality generated by a computer. Here, the user is no longer a passive consumer of someone else’s database. By exploring and interacting with the environment, she orients herself according to her own interests. One might even go so far as to say that she becomes a kind of film director framing close-ups and long-shots.

Philosopher Vilém Flusser has characterized the time we live in as an epistemic crisis brought about by the recognition that, “scientific research is not the gesture of a transcendent intellect.”  What Flusser calls “the gesture of searching,” for meaning or knowledge, has shifted from “a digging down for reasons” to an examination of the aesthetic and relational qualities of things and an exploration of how humans respond and attend to their environments as subjects and as objects (Flusser, 2014, “The Gesture of Searching”). Perhaps unsurprisingly, Flusser’s description of this gesture serves as an apt description of being in virtual reality, a reality which has become, in essence, a database.

…the researcher is embedded in an environment that interests (matters to) him, both at close range and at a distance. There are aspects of the environment that interest him intensely and others that hardly touch him. The more an aspect of the environment interests the researcher, the more ‘real’ it is for him (Flusser, 2014, “The Gesture of Searching).

In this essay, I will build upon Manovich’s observation that, “cinema language, which was originally an interface to narrative taking place in 3-D space, is now becoming an interface to all types of computer data and media” (Manovich, 2001, 326). I will argue that whatever the language of VR turns out to be, it will rely far less on symbols and signs (content)  and more on a procedural code (form) that elicits both the memory of the body’s movements and affects as well as a shared cultural memory, not of the lived world, but of cinema, itself.

What determines the authenticity or veracity of this kind of storytelling? Here, it is useful to consider one of the earliest, most widely distributed experiences in VR. The Tuscan Villa  was provided free with the developer kit for the Oculus Rift DK2 head mounted display. It offers users the chance to meander through a country villa and surrounding landscape. A generic looking computer-generated farm house, its stones visually coded as “old,” suffices to invoke a nostalgic memory—one that will be even more poignant and believable to a viewer who has never actually stayed in a Tuscan villa. In other words, the Tuscan villa is meant to exist outside of history. In this work, the visual and auditory codes are perceived as “authentic” if they successfully activate a process of false memory and effect a change in the viewer’s affective state. Significantly, the experience does not provide a traditional narrative. Rather, it  offers the vague sense of remembering a dream or retrieving something that has been lost which never actually existed.

Kitsch is nothing if not a suspended memory whose elusiveness is made ever more keen by its extreme iconicity […] kitsch is not an active commodity naively infused with a wish image, but rather a failed commodity […] a virtual image, existing in the impossibility of fully being. Here, for a second or even a few minutes, there reigns an illusion of completeness, a universe devoid of past and future […]. (Olalquiaga, 1998, 28).

In The Language of New Media, Manovich points out that as cinema becomes more and more reliant on software manipulation in postproduction, it functions less as “a record of perception,” and more like a computer screen— “as a record of memory” (Manovich, 2001, 325). In the case of the Tuscan Villa, cinema no longer serves an indexical function vis a vis reality, but rather, it registers the trace of a shared cinematic memory. The kitsch experience of the villa and garden offers a momentary illusion or simulacrum of “realness”. For this, it sacrifices the passage of time and complexity of meaning.

Flusser suggests that gesture, by which he means the “reality” of affective states made manifest through aesthetics, should be judged in terms of art or kitsch rather than whether the gesture is true or false. “The scale of values we use to evaluate (gesture) may not oscillate between truth and error or between truth and lies but must move between truth (authenticity) and kitsch (Flusser, 2014, The Gesture of Searching).

It is not artifice in itself that renders the Tuscan villa demo artless. To understand this,  it is instructive to compare the Oculus demo to Maria Menken’s  decidedly un-kitsch 1957 film Glimpse of the Garden . Even though both The Tuscan Villa demo and Menken’s film privilege surface effects over depth of meaning, the result is decidedly different. In her film, movement of the camera, a unifying looped soundtrack of birdsong, and color filters take precedence over content, which consists mostly of shots of cultivated nature. Despite the amateurish quality of the images, the result is strangely magical. Menken’s editing, which juxtaposes different scales, points of view, and speeds of movement, resists the Tuscan villa illusion of completeness. Thus, it is due to the authenticity of the filmmaking gesture itself, not its content, that Menken’s film can be called art whereas the Oculus demo remains kitsch.

If there is any redeeming gesture in the Villa, it is in its ability to invoke an uncanny sense of bodily presence, a being there in a digital world that is generated by the user’s wandering or, in Flusser’s terminology, “searching” through the environment. According to neuroscientists,  presence is a primordial mechanism by which organisms establish self and other, creates a motor map of the universe and uses memory to plan actions and interactions. In so far as the Tuscan Villa succeeds in creating this sense of presence, it sets up a queer dynamic in the user—an oscillation between interior and exterior, self and other, the “real” and the virtual. In this way it, perhaps inadvertently, succeeds in making the familiar strange. In other words, the odd and unresolvable experience of being in two places at once creates a kind of digital estrangement conceptually related to Bertolt Brecht’s Verfremdungseffekt, distancing effect, and Russian Formalist critic Viktor Shklovsky’s “making strange” (Russian: приём остранения priyom ostraneniya) whereby certain devices are used to disrupt the audience’s easy identification with an artwork.

To see how this might be productively used in VR, I turn to filmmaker Karen Cytter’s work for a contemporary example of estrangement. Cytter employs familiar, even kitsch codes of cinema, especially melodrama. But, rather than offer an illusion of completeness, she uses what Thomas LaMarre refers to as “internal montage” (LaMarre, 2009,125). LaMarre suggests that computer graphic imagery and digital effects have created a new emphasis on the animation technique of compositing —the creation of an image composed of multiple layers. If an artist suppresses the space between layers to create an illusion of wholeness, the result is either a unification in depth, a hyper-Cartesianism with a well defined vanishing point (e.g. Pixar type animations), or a flattening of all layers into a single “superflat” plane in which multiple frames of reference and sight lines coexist. Cytter’s films demonstrate a remarkable compositing of space, dramatic events, and sound. It is effectively a jumbled cinematic database in which voice, visual form, emotion, and plot nearly float free of each other on the level of meaning.

Surface effects are at least as important as content here. Through a process of making the familiar strange, she astutely reveals the underlying game of seduction that takes place in cinema. In Cytter’s work we are left with empty codes, empty signs which nevertheless incite emotional responses. Devoid of linear narrative, in Rose Garden , the mournful sound of the flute is not only a wry reference to Sergio Leone’s Once Upon A Time in the West, it codes the psychological tenor of the piece. Although there are no clear “characters,” as with avatars in video games, the human body remains the basis for emotional engagement. For instance, at the end of the film, when a boy is shot in the back, though we know nothing about him as a character, the violent collapse of a child’s body is still shocking and emotionally wrenching.

Rose Garden’s flattening out of time and space, and a move from depth to surface, which Flusser relates to the gesture of searching, resonates with Hiroki Azuma’s description of database narrativity. In his book, Otaku: Japan’s Database Animals, he notes a move from “grand narratives” to “grand non-narratives” or database (Azuma, 2009, 55). This is associated with the proliferation of small narratives the purpose of which is not to refer back to some greater or universal theme, but to invoke strong feeling or fulfill consumer preferences. Whereas, in the modernist era, personal fictions could be related in depth to a grand story about the nature or meaning of human life, now, there is a lateral move in which multiple small stories are spawned from the same database as in Cytter’s film. Likewise, in virtual reality, narrative unfolds as movements through space. However, distance is no longer measured in terms of meters or any other universal objective standard, but, as both Flusser and Azuma suggest, in terms of the user’s desires and interests interacting with the environment.

In Mark Amerika’s cell phone film Immobilité, one of the characters remarks that the world is disappearing before her eyes. Indeed, the opening scenes, a shaky, looped walk down a tree-canopied path, reveal just how the world will end— not in some fiery apocalypse, but rather by our recognition that the world was only ever virtual. In Immobilité, what remains of embodied experience, what Amerika calls “muscle memory,” finds its expression not in meaning, but in form. In effect, Immobilité — a film that remixes some of Flusser’s thought into the subtitle track — consists not so much as a series of images, but as a series of film gestures.

In Immobilité, the quality of the image is sacrificed to movement of the camera just as content is sacrificed to surface effects. Depth, in the form of philosophic musings, is separated from the images and brought to the surface as subtitle. What remains on the screen are vibrations, rhythms, and frequencies that resonate with or cancel each other out. In fact, in many parts of the film, the camera is decidedly unsteady or even appears to vibrate thereby creating a surface that destroys a Cartesian perspective and the static perceiving subject. “And then I would begin to lose myself, to play uncontrollably, becoming something like music,“ the unreliable narrator texts us from the beyond.

Heidegger’s concept of “profound boredom” operates here. This is state in which all beings “recede” into indifference and the one who is bored comes face to face with the experience of time itself (Heidegger, 2001, 80). Heidegger describes profound boredom as the ground for all the multifarious variations of being. Profound boredom leads us to suspect that reality is neither static (immobile) nor completely knowable. Concomitant with this loss of a fixed self, language fails to signify. Rather, it functions procedurally to carve up pure duration into pieces: life and death, existence and nonexistence, chaos and order. This resonates with Henri Bergson’s suggestion in Matter and Memory that homogeneous and universal perceptions of space and time are a trick of the mind, a refraction of pure duration into space. For Bergson, space and time do not exist anterior to perception and action. We do not act through space and time, rather we create these symbolic diagrams because we need to “divide the continuous, fix the becoming, and provide our activity with points to which it can be applied (Bergson, 1990. 212). Bergson suggests that our knowledge of the world is delimited by the concerns of the body and its possible actions and, perhaps, even more importantly, by our memories. If that is true, we are like Plato’s cave dwellers: always living in a virtual reality of our own making.

The key dynamic in Amerika’s film, one that will likely dominate in this time of apocalyptic narrative, is a tension between overt aesthetics and conventional notions of truth. Although Immobilité’s low production values and distinct lack of artful montage lend the film a “real-life” documentary feel, Amerika’s overt use of glitch, repetition, and jerky camera movement insists upon a kind of self-conscious performativity or aesthetics. When the narrator asks, “Was I authentic?” — the question is equally what does “authentic” mean if everything is virtual and, ultimately, data, and, furthermore, who is “I”? If the world is disappearing, so is the singular author, that figure of the auteur which haunts French New Wave cinema, to which Amerika  pays homage in his use of subtitles. In Immobilité’,  Amerika, like Flusser, suggests that the question of “authenticity” lies not in a “true” identity but in the aesthetic value of the gesture itself.

Antonioni’s masterwork, Zabriskie Point, also operates by disavowing conventional notions of authenticity. In Zabriskie Point all is surface: from the setting, an almost abstract stretch of ancient desert, to the way images are composed in multiple layers without depth. There is also a strange flatness to the main characters. They seem to have no ambitions and no particular future. With exquisite conciseness, Antonioni destroys the epic march of history with the first line spoken by the male protagonist: “I’m willing to die (for the revolution), but not out of boredom.” For the characters in Zabriskie Point,  time operates as it does in profound boredom as a pure duration.

Even on a personal level, events have no apparent consequence. Though one of the young lovers dies violently, there seems to be little emotional repercussion for the lover that remains. This flatness of character is matched by the physical beauty of the actors. It does not matter that beneath they are voids, the viewer can still be seduced into watching. In the end, it is up to the viewer to decide if Zabriskie Point’s aesthetics, which privilege form over content, offer any meaning.

The last scene of the film, which is overtly apocalyptic, masterfully captures this dynamic. Here, the female lover imagines blowing up the ultra-modern house of her boss, a slick real estate developer who wants to produce a housing development in the desert. Antonioni’s slow motion explosion which shows mostly intact objects floating across a horizonless, flat blue surface of sky disrupts conventional Cartesian perspective as well as scale. Moreover, using slow motion and repeating the moment of the explosion from multiple points of view, Antonioni disrupts the conventional flow of time and a singular perceiving subject. However, as with the rest of the movie, nothing actually transpires. The film concludes with the woman, a virtual terrorist, driving away from the intact house without a clear destination.

The strange physics of Antonioni’s explosion brings to mind the procedural physics operative in the software used to create VR experiences. These are the rules by which objects behave in a virtual 3D landscape, rules of interaction put into play by an algorithm in a game engine. As Baudrillard notes in his odd little book Seduction, games are subversive precisely because they allow for a reversibility and fluidity of meaning that operates outside of law (Baudrillard, 1991,131-157). Because procedural physics operates outside of natural law,  it offers the potential for radically new forms of narrative.

As the quote below from Donald MacKay, one of the key figures at the Macy Conferences on Cybernetics, points out: meaning, itself, acts as an “organizing function” for a perceiving subject. Significantly, MacKay defines information according to its functional impact, without respect to whether it is “true” or “false,”  an idea which resonates with Flusser’s insistence that aesthetics determines the truth of a gesture.

INFORMATION can now be defined as that which does logical work on the organism’s orientation (whether correctly or not, and whether by adding to, replacing, or confirming the functional linkages of the orienting system). Thus, we leave open the question whether the information is true, false, fresh, corrective or confirmatory, and so on…The MEANING of an indicative item of information to the organism may now be defined as its selective function on the range of the organism’s possible states of orientation (Wilden, 1982, 236).

Discussing filmmaker Peter Greenaway’s move from film to installation, Lev Manovich suggests that spatialization allows Greenaway to construct pure database narrative, something which film does not allow.

No longer having to conform to the linear medium of film, the elements of a database are spatialized within a museum or even the whole city. This move can be read as the desire to create a database at its most pure form: the set of elements not ordered in any way. If the elements exist in one dimension (time of a film, list on a page), they will be inevitably ordered. So the only way to create a pure database is to spatialize it, distributing the elements in space. (Manovich, “Database as a Symbolic Form”).

Following Bergson’s emphasis on the body as the starting point for perception, I would counter that this does not constitute pure database. At least in the “real world,” the narrative structure of installation arises out of the interaction of bodies with the architectural environment. As Lakoff and Johnson, and others have shown, human language, also revolves around the body. However, in the virtual world, this is not necessarily so. Here, even gravity need not operate according to natural law and, furthermore, reading text can often induce nausea. Yet, even in virtual reality, meaning-making requires some form of structure.

One way that virtual reality narratives might recoup meaning without resorting to conventional cause and effect is by using procedural physics to structure narrative and to create asymmetries in the environment (database) which allow for meaning-making according to MacKay’s definition.

Artist Rachel Rossin’s virtual reality works illustrate this possibility of a new kind of narrative, one that speaks to and through virtualized, cyborg bodies. In an early work n=7/The Wake in Heat of Collapse, Rossin uses the procedural physics of gravity itself to shape narrative. Here, users progress through three contemporary Dantesque landscapes. Gravity in the landscape decreases algorithmically as the user descends. Invoking the virtual body of the user, Rossin notes:

In video games you have a certain type of hermeneutics…what you understand as the sort of “body language” or instinct that come with the world. Gravity – especially – falling off of things usually means death. We get this sad little feeling when we watch ourselves die in a video game and there is a pause that actually feels like critical distance and so I used that as a loose Virgil… the thing that pulls you through the layers of the separate worlds (Rossin, personal email correspondence December, 2015).

In a more recent work, I Came and Went as a Ghost Hand, Rossin scanned intimate photos of her domestic surroundings then manipulated these and used them to create a virtual reality landscape. As a result, the viewer sees familiar things differently, not just because of the data loss, which reduces some objects to their raw qualities: color, shape, volume, and which causes erasures, but because of the unsettling experience itself: moving through objects, speeding up, slowing down, torque. In lieu of a narrator, Rossin offers a reassuringly banal, reoccurring icon of a white gloved hand, a kitsch gesture that acts as a point of continuity for viewers as they traverse the disorienting, horizonless landscape.

In his essay, “Envisioning the Virtual,”  Brian Massumi suggests that the virtual should be considered as the formative or potential dimension of reality, not as something which is in opposition to actuality. Analyzing an optical illusion where “Pac-man” circles cause a virtual triangle to appear, Massumi proposes that it is the tension between different forms of perception which produces “a field of intensity” within which the “pressure for resolution” acts as “a formative force.”

The tensions are between modes of existence proposing themselves to the experience[…] The modes do not add up to a form. They are tensely, incommensurably different. Their incommensurability exerts a differential pressure. Something has to give (Massumi, 2014, 56).

Massumi is describing a situation in which different ways of looking at the world seek to coexist in time. Cinema attempted to resolve this with montage, VR offers a different solution: the spatialization of time itself. How humans will organize that time-space in a meaningful way with respect to virtual bodies is evolving. Moreover, the political and social implications of this are as yet unknown. However, as I have shown, the memory of the real, preserved as a visual code borrowed from shared cinematic experience and the memory of the body, preserved as codified gesture will likely dominate the language of VR, at least for now. What gives way after that, what new perceptions and new narrative forms might be revealed and uncovered with the aid of human-computer interfaces remains to be seen. 

Bibliography

AZUMA, Hiroki, Otaku: Japan’s Database Animals, translated from the Japanese by Jonathan E. Abel and Shion Kono, Minneapolis (MN), University of Minnesota Press, 2009.

BAUDRILLARD, Jean, Seduction, translated from the French by Brian Singer, New York, Palgrave Macmilan,1991.

BERGSON, Henri, Matter and Memory, translated from the French by N.M. Paul and W.S. Palmer, Zone Books, Cambridge (MA), MIT Press, 1990.

FLUSSER, Vilém, Gestures, Kindle edition, translated from the German by Nancy Ann Roth, Minneapolis (MN), University of Minnesota Press, 2014.

HEIDEGGER, Martin, The Fundamental Concepts of Metaphysics: World, Finitude, Solitude, translated from the German by William McNeill and Nicholas Walker, Bloomington (IN), Indiana University Press, 2001.

LAKOFF, George and Mark JOHNSON, Metaphors We Live By, Chicago (IL), University of Chicago Press, 2008.

LAMARRE, Thomas, The Anime Machine: A Media Theory of Animation, Minneapolis (MN), University of Minnesota Press, 2009.

MANOVICH, Lev, “Database as a Symbolic Form,” http://www.mfj-online.org/journalPages/MFJ34/Manovich_Database_FrameSet.html, accessed online 22/01/2017.

———, The Language of New Media, Cambridge (MA), MIT Press, 2001.

MASSUMI, Brian, “Envisioning the Virtual,” Oxford Handbook of Virtuality, Oxford, Oxford University Press, 2014.

OLALQUIAGA, Celeste, The Artificial Kingdom: On the Kitsch Experience, Minneapolis (MN), University of Minnesota Press, 2002.

WILDEN, Anthony,  System and Structure: Essays in Communication and Exchange, New York, Tavistock Publications, 1980.