Here is my talk on XR and Theater. If you are interested in discussing further, please get in touch! I love thinking! SLIDES

SLIDE 1: I’m Illya Szilak. I’m a writer, artist, director and also, a medical doctor.

SLIDE 2 : I’m going to start with a little movie from a hundred years ago. This is really the beginning of film as an art form. In this film, which is only about 2 min long, you will see what looks like a cosmos, generated by Man Ray be sprinkling salt and pepper on film and it will end with a body, of sorts. Man Ray was a surrealist. To some extent, World War 1, change everything. Suddenly nothing made sense and human progress, those enlightenment ideals of reason, were shown to be if not utterly bankrupt, not easily achieved. So, Man Ray made this. As you watch this, I want you to think about who or what is the actor? How does “liveness”, the quality of being alive and having agency arise in this work? And, is it reason that allows you to make sense of what you see? What organizes and connects information in the scenes of the film? Is the film as a whole a kind of performer? These questions of performativity and liveness are ones to which we will return.

PLAY SLIDE 2

SLIDE 3

To begin this talk, I am going to start with some very basic assumptions that I hope you will agree with. First, is that storytelling is perhaps the quintessentially human way of organizing information. Second, technology allows us to access information that is otherwise inaccessible to us through our senses. Third, the repercussions of this are not entirely benign. • In a world where information is being created so quickly, as you see on this slide, most of it unstructured that is to say it’s not part of a predefined data set or model, the critical question is how do we choose what information to pay attention to and value? How much does technology itself determine what kinds of information we pay attention to and value?

In this talk, I’m going to delve into some philosophical and neuroscientific perspectives on this. I’ll argue that although all technologies from language on up have influenced how humans see and organize their world, the impact of computer technologies is unprecedented. I will argue that in this Anthropocene age, we must find a way to relate to each other and to the planet differently if we are to survive and thrive as technologically extended humans. Theater is a way of playing out, rehearsing ways of being in the world and also seeing ourselves through the technologies that we use. XR Theater is especially powerful as it allows us to inhabit two positions, self and other at once. It allows us to play with new forms of liveness, and in a lived, as opposed to purely theoretical way,  explore what we machine extended beings should value. Do we value the body? Do we value gravity and work (in all its meanings). Which ways of perceiving time and space do we privilege? How do we represent ourselves? How do we relate to, represent, and acknowledge (or not) gender, race, age, species, ability in machine extended environments.

To start, I’m going to take you on a short foray into how philosophers of the machine age have thought about how we organize information. 

SLIDE 4: BERGSON
We won’t start too far back. Let’s start when media technology really started to take off telephone, photography, phonograph, film. The year is 1894. This is when French philosopher Henri Bergson published an amazing book which all of you should read. In Matter and Memory, he suggests that the universe is made up of an infinite number of images: sound images, thought images, visual images, etc. and the one’s that we pay attention to are ones related to two things: our bodies and memory. 

SLIDE 5: UXEKILL
Jump ahead thirty years and we have Jacob von Uxekill publishes a little book in which he creates a thought experiment imagining how the world to different species including the deaf and blind tick which perceives the world in terms of and responds to levels of butyric acid radiating off of a mammal’s skin. These worlds he referred to as “Umvelt” a lived-world of experience.

SLIDE 6: MCLUHAN
Now jump to 1967 where Marshall McLuhan coins the phrase “the medium is the message,” arguing that content per se could not be separated from the medium. Media acts as a kind of sense organ for humans, how we now construct our “Umwelt” is not simply through our native senses but through all our media extended senses. One might even argue that with most of us living through our devices much of the time, the human “Umwelt” has undergone an apocalyptic change.

SLIDE 6: VARELA
A century after Bergson, Francisco Varela and his colleagues published this book which created a revolution in thinking about how humans process information. They argued that the idea that human reason3Zxq ZXCGVHJIL;’’DF could some how be separated from the body, an idea which really took hold during the Enlightenment and became valorized with the advent of computing machines, was just plain wrong. We don’t just passively receive information from out there and translate these into abstract symbols which we then translate into actions. We enact our world.

SLIDE 9

Moving on to 2000, philosopher Villem Flusser argues for something even more provocative, technology , he suggests, doesn’t just alter the message, as McLuhan suggests, it alters the way human consciousness works, the way we organize information.  The ‘apparatus’ — the camera is a tool has its own ‘program,” one which supercedes human control. Photographers essentially are playing a kind of game according to the camera’s preprogrammed rules. It is a menu of choices that arise from the apparatus itself and not from within the human.

SLIDE 10:  The issue is that the images no longer help us orient ourselves into the world because their basis is an exploration of the program and potential of the apparatus, not of the lived world itself. Covid was the straw that broke all our backs, because it showed us that amidst that chaos of images, there is one that can not be captured. That is death. This is the fact: all of of us will get sick and die. Covid showed us this and it scared the shit out of us. Because we could no longer be distracted from the one reality that is beyond the reach of our technologies. Given that our computer technologies do not privilege embodiment but rather embodiment as captured and experience through image, one that removed from the evanescent now and placed into a kind of undead forever now, the question we were starkly left with is how does matter matter?

The problem of the apparatus is that human cognition is not just an abstraction, it is not an array of menu choices, it is a lived, embodied process that has evolved over millions of years. The question is whether technology is impoverishing human consciousness or enriching it. Yes, there is more information, but perhaps because of our estrangement from our lived worlds, worlds not translated through the menu of an apparatus, that only leaves us in chaos?

SLIDE 11:
It does leave us in chaos. Let’s go back to 1951. The Macy Conferences where information theory was born. What is information? On one side we had Claude Shannon, brilliant mathematician and computer scientist. He wanted to find a definition for information that would be quantifiable and something machines could understand.

The best way to illustrate Shannon’s definition of information is that the flip of a double headed quarter contains no information because there is only one possibility. Physicist Donald Mackay disagreed. He argued that meaning acts as an organizing function for a perceiving subject, it had an impact, it was not merely related to the number of bytes.

SLIDE 12
But, Shannon’s formulation, the efficiency of the code, the losslessness of transmission won out over MacKay’s meaning-making. As N. Kathryn Hayles points out, Shannon’s definition of information is essentially a disembodied form of information. So, what we are left with is a situation where human cognition is embodied, but our Umwelt as constructed by our technologies is not. And, We have no idea what this means.

But, consider for a moment what it might mean in terms of how we relate to our planet and our fellow humans and other sentient beings. I don’t have to put up graphs about the rates of opiate overdose, depression, suicide, mass shootings, species annihilation, ecosystem destruction to make a case for why we need to figure out new ways of being in the world and with each other and fast. The problem is that human beings have trouble with chaos, it makes us really nervous. So, as the totality of information increases and the probability of any one answer decreases, well, we got trouble. Plato kicked the lyric poets out of the polis for a reason. Fascism, religious fundamentalism, and, weapons of mass destruction or just down-home schoolyard gun massacres well these are all understandable responses.

Although this information explosion and concomitant disorientation, which honestly, is this close to insanity can be and will be and is being placated for a time through the dispersal of this tension through the pure act of repetition and compulsion (our swipes, our constant checking of emails and circulation of memes and data), our fear of annihilation by distributing ourselves in multiple identities across virtual spaces, a withdrawal from attempts at complex meaning-making to Tweet size bites, 10 second TikTok videos and Pixar like bright and shiny emotions. Even so, the center will not hold forever. Something is going to break.

SLIDE 13
There is no going back, return to the days of unextended embodiment. The horrible wonderful fact is that our ability to extend ourselves in space-time through machines has brought us to a critical moment where the death drive and Eros meet. Metamorphosis is to acquire a different border between self/other, a different form, different senses, different ways of relating to , privileging and processing information and our relationship to other living beings and to our home, this planet.As marabou suggests, Metamorphosis is what occurs when fleeing is necessary but impossible. And, it is what we need now.

SLIDE 14
Apologies to Elon Musk, your flight to Mars, yes, I understand it on a human level of terror and imminent annihilation, but, sweetheart, it ain’t going to work. And, by the way Twitter or X or whatever, probably not helping things… The bottom line is we must find a way to relate to each other and to the planet differently, because not everyone is gonna be on that spaceship.

SLIDE 15 HARAWAY
As such, it is critical that artists stake a claim and use new technologies to interrogate the relationship between brutal historical, political and material realities and the pernicious “correction” of these facts, not only through omission e.g. Facebook Horizons, but also, and perhaps more importantly through “neutral” software based regulation of identity, behavior, interaction and access to and production of information in digital spaces. As Haraway suggests: the relation between organism and machine has been a border war. The stakes in the border war have been the territories of production, reproduction, and imagination. And, I would suggest, survival.

Where will this border war take place except in our digital spaces? One could attempt to hack dominant technological systems. It has been done. But, it’s probably unlikely that you will bring Twitter or X or whatever down for long, the question then becomes one of finding ways to be, interact and live differently inside those dominant technologies, that is to say, we must begin to play with and in them. We must not merely be users, but players.

SLIDE

So, that brings us to theater. In 1924, Italian Futurist F.T. Marinetti comes up with the idea for “total theater.” According to scholar Pierpaolo Antonello, “Marinetti saw in the theatre one domain where art could intervene to radically alter the cognitive and perceptive experience of the spectator.” Marinetti’s productions were not script-based, but multi-sensorial and performative. Among the senses, Marinetti privileged the sense of touch. 

Marinetti’s Manifesto of Tactilism published in that year, is certainly worth a read in its own right, it is hyperbolic and poetic, but especially fascinating for me is this last paragraph which posits the possibility that there could be other senses, other kinds of cognitive inputs, beyond language and our five senses. I think what Marinetti is actually getting at is the possibility that a work of art might be able to elicit these new ways of experiencing the world. In other words, an artwork might enable us to overcome the conventional limits of how we perceive the world, not merely conceptually, but also materially as a lived experience.  

SLIDE 17
Marinetti anticipated the possibility of total theater a century before that technology became widely available (thanks Apple!). So why not just immersive theater as Marinetti imagined it? Why XR. The power of XR with regards to total theater has to do with two factors. First is the ability of XR especially VR to generate a remarkable sense of presence—That sensation of being here and now in time and space that seems unmediated. There are many theories of presence in XR but the one that I like from Riva and colleagues draws upon the work of
neuroscientist Antonio Damasio. Riva suggests that presence is a primordial form of consciousness, one that helps animals distinguish self from other, and internal from external stimuli. Interactivity especially action upon the world is critical to facilitating this. Through interactivity, an organism creates a motor-map of the universe. This allows it to plan future actions in the light of past knowledge.

SLIDE 18
This slide outlines Damasio’s description of the creation of the self, based on this, Riva describes three forms of presence. Proto presence is an organism’s ability to couple perceptions and movements and thus differentiate itself from the external world. Core presence is an organism’s ability to focus sensorial experience on present tasks and relevant objects in the environment. Extended presence is an organism’s ability to recognize the significance of previous experiences as they relate to the self. So in this last personal, historical, aesthetic and cultural memory play an important role.

So, if presence and self creation are linked, it’s clear that the power of XR for playing with new versions of being and being in the world is that the self of the audience occupies two positions in space and time. This duality may be formally acknowledged through the use of an avatar, or may be inferred by the visitor through the architecting of two or more space-times within the XR experience.

SLIDE 19
In XR, meaning-making in XR theater occurs in the oscillation between two registers within the audience/participant. This relates to the dual nature of self as embodied self and as avatar/other that XR encourages.On the one hand there is the sensual self the enacts an environment from the five senses, on the other hand there is a symbolic, representational self. Clearly the former is concerned with affect generated by action, the latter is concerned with how the self is represented which can be recognized on a cultural, historical and aesthetic level.
In this way, XR theater offers the possibility of a new kind of storytelling, an active, enacted kind of storytelling—This language relies far less on symbols and signs, and more on code that elicits both the muscle memory of conventionally embodied sensations and the emergence of new ones. In such a language, affect and narrative results procedurally, through form and process even more than through content. The new form of being elicited in XR theater oscillates between self and other, material and virtual body, life and art, audience and performer. XR Theater is a laboratory for this metamorphosis.

SLIDE 20
In this slide I want to point out some of the dramatic tensions which operate generally in XR theater. These do not necessarily take the place of conventional elements of dramatic tension such as opposing characters, and dramatic action, but they are important to consider. I don’t have time to go through all these and I’m sure there are others, but I’d like to highlight a couple of these.

The first is thinking about the relationship between embodied material reality and the transcendent body interrelate. Two other that I would like to highlight is the relationship between the phenomenologically situated body and the character or symbolic body most commonly this might be experienced as an avatar. Now, not only do people project bodily action onto an avatar, such that the action of the avatar body overrides the orientation of the physical body, but how that body is represented becomes important. To represent an avatar body is anime form and using volumetric video are not the same. There are situations when the affordances of one are more useful. I’d also like to point out that aesthetics become a hallmark of authenticity in these worlds where everything is unreal. This is why art matters. Now, perhaps, more than it ever has. Technical images can not be judged as true or false but only as art or kitsch. Unfortunately, but not surprisingly, the tech industry privileges hyperrealism and Pixar seductiveness above all other aesthetics and emotional registers. AND< we love it! Of course we do, who could blame us? Like Elon Musk, we are fucking terrified. Kitsch is mother’s milk. In its embrace we can distance ourselves from chaos through irony or come together so we can all cry Milan Kundera’s second tear together. The memory of the “real”, preserved as codes borrowed from shared cultural experience and the memory of the body, preserved as Kitsch aesthetic and genre and gesture will likely dominate the language of VR, at least for now. What gives way after that, what new perceptions and new narrative forms 
might be revealed and uncovered with the aid of human-computer interfaces 
remains to be seen. 

SLIDE 21

Because, the point of XR theater is not to reconstitute traditional theater in the digital space, but to utilize the the affordances of technology to allow the art form to evolve in a way that speaks to and through machine extended bodies.. XR theater may not look like conventional theater. In fact, not only does it take its inspiration from multiple arts involved with spatial construction and movement, it may, take on the form of any of these.

SLIDE 22
Understand that in XR theater, the environment, the objects in that environment, and the virtual or other humans in the play all perform. This is not new. We saw this in Man Ray’s film, that nails can “dance.” Is this liveness? No, it is a simulacrum of liveness which, as we know from the concept of aesthetic sympathy, the original meaning of empathy, nevertheless constitutes a way for us, the audience, to experience our own liveness, our own bodies dancing like nails. The word empathy which psychologist Edward Titchener coined in the early 1900’s, did not mean what it means today, rather it was seen as a form of aesthetic sympathy, one which allowed audiences to appreciate the the abstract movements of modern dance and painting which were just emerging. Here is a beautiful evocation of that empathy.

SLIDE 23 LIVENESS

Since, in XR both environment and objects and virtual humans are responsive and respond, what constitutes liveness in XR theater? I do not have time to go into the debate represented in these quotes, but my own take is that our experience of “liveness” manifests on many levels, some more familiar and proximate than others. It is certainly not limited to conventional live performance. As long as the audience themselves is conscious, that is what is required. A live audience member brings with them a sense of self and a sense of being changed or wanting to be changed or touched in some way by the experience. So without getting into a debate on the nature of consciousness, I’d say that liveness brought to the work of theater by the audience relates to her awareness, her sense of self/other and to desire. Her consciousness is the spark that energizes the whole machine bringing it to life. In this way, the work of art itself is a kind of cyborg entity with manifestation of that liveness operating on multiple registers.

SLIDE 23
XR theater should explore new forms of “materiality” which I list here. To illustrate some of these concepts, I am going to dive into our newest project FLY ANGEL SOUL supported by the Jerome Foundation. Fly Angel Soul is a short experimental narrative film shot within virtual reality. It tells the story of Sebastian, a young gay physician estranged from his rural Catholic Missouri family, who, having moved to Mali to heal the sick, is diagnosed with AIDS. Inspired by a quote from Meister Eckhart “(let us) rejoice in the everlasting truth in which the highest angel and the soul and the fly are equal,” Fly Angel Soul is shot in real-time, from the unique points of view of three networked virtual cameras adopting the “roles” of the eponymous characters. The “human” p.o.v. will be that of a live cinematographer moving through the virtual set. Thus, in Fly Angel Soul, “liveness” resides in the “embodied” cameras even more so than in the actors in the story. Finding commonality with video games and live performance, Fly Angel Soul explores the potential for virtual production techniques to expand 2-D cinematic language. The human camera viewpoint which predominates in this first scene, what we affectionately refer to as the soul cam, is virtual as are the angel cam and the fly cam. Created by an amazing developer and Unity engineer named Elliott Mitchell. But, unlike the angel and the fly cam, an actual human, dancer, choreographer and creative technologist Clemence DeBaig created the path that the soul camera follows through the virtual mis-en-scene in Cinemachine.

SLIDE 24
Our goal here was to use our experience and skills honed in creating interactive stories and installations to create a film which put the film audience in a position of choice. Whether it is conscious or not, each viewer will have a preference for one or other of these perspectives. We have already found this to be the case among the film’s own team members. This points out not only the agency of the viewer, but also their responsibility. Since the world can be viewed in multiple ways, something both art and technology show us, then, how we choose to view it and how we choose to tell a story, what information and perspectives we privilege depend on what we value. There is no singular truth. The old gravitational centers of God and tradition are gone or fading fast, this is beauty and terror wrought by our technologies. But, amidst all this, there is one truth that few of us can see differently, one which Covid made us acutely remember, we are all of us mortal. We will all get sick or injured and die. This is what this story is about, the imperative to find new ways of relating to ourselves and each other. I have an MD after my name and I still practice medicine actually at Rikers Island in New York City, so my relationship to mortality and embodiment is perhaps less abstract than most people in tech. It was of critical interest to us that we find a way to use technology to make the viewer feel as though they were really intimately present in the scenes.

SLIDE 25
There were cinematic inspirations for this especially the long tracking shots in Max Ophuls films and Birdman. And, the use of gesture and architecture in Maya Deren’s films.

PLAY SIDE 26
I’m going to play a little bit from Birdman. Notice the architecture, notice the multiple rhythms of movement including sound, camera movement and actor movement, how this unites everything into an incredibly immersive experience.

SLIDE 27
First I want to let you know that I have never actually made a film before, so although I know we completely upended the usual workflows for filmmaking, it didn’t actually seem weird to us. Once I had a script, we began to work on what I call a responsive mis-en-scene: this was a responsive environment including architecture, spatial sound, and movement based triggers. To facilitate our collective imaginings, my artistic partner Cyril Tsiboulski, created a quick mock up in VR Chat from the initial architectural sketches that artist/architect Paolo Barlascini had come up with. We actually met there with our actors and rehearsed. Here, some of us are in VR Chat.

PLAY video

SLIDE 28

Aesthetically, we were interested in finding a sweet spot between the real and the transcendent, between intimacy and overt theatricality. For instance, the doctor’s clinic is an impossible structure. On the outside, it references Malian mud architecture, on the inside it references the radial architecture of the original plan for St. Paul’s Basilica in Rome. This is obviously possible only in VR. We could have created everything virtually, but we really wanted the hand of the artist to be present in the form of mark making, a trace of the artist’s performance. So Paolo painted and drew and then scanned these and working with developer Emmy Yupa, these paintings and drawings became the skins for the walls of the interior of the clinic. In addition, we worked with the sound designers so that the soundtrack could be responsive to the movement of the human camera and robust enough to sound great each time. Because, this film can be reshot and reshot. Lastly, we worked with Clem on movements for the actors. Mostly the second scene is literally waiting. The dialogue is created in units with breaks and movement sequences that can be triggered when the soul cam moves away. So again, we are creating a tension between intimacy and theatricality.

SLIDE 29

This intimacy, the liveness of the film comes not from the actors performances, these are essentially artifacts of a performance. They were filmed at the Microsoft mixed reality studio in San Francisco (unfortunately, now closed) and are essentially game objects. We used volumetric as opposed to Mocap or animation because we really think that volumetric not only captures the nuances of human movement, but it registers as a trace of the living, the real, it feels like an authentic memory of something that happened. But, the liveness of the film comes from the living camera and the misenscene that responds in real time to her presence.

SLIDE 30
This May we shot the soul cam path at the NYU Tandon Mixed Reality Lab. The first day was taken up by hardware malfunctions. The second day was revelatory. We literally had no idea whether this crazy project we were working on would work. But, after allowing for a discovery phase, I asked Clem to pretend that she was Sebastian’s estranged mother who finds herself in a dream. How would she move through the landscape, how would she move in relation to him. It was a revelation. I went back and forth between watching Clem moving in real time in headset to watching her move virtually through the misenscene. It was magic.

SLIDE 31
I’m going to play a little of this film for you, it is still a draft.

PLAY FILM

I hope it shows you that XR theater can take many forms. Our goal after this is to find funding for what we call real-time filmmaking. This would combine live theatrical performance and live-streaming of FLY ANGEL SOUL made there on the spot. If you miss it, the film never existed. Filmmaking itself becomes performance.