CannesXR Keynote by Sarah Vick, executive producer, Queerskins ARK in conversation with me about Volumetric video, movement, and empathy in VR

Although a growing body of research shows that “bodily postures, gestures and expressions are inherent components of emotional experience and tacitly influence the evaluation of persons, objects and situations as well as memory recall,” researchers have yet to fully exploit these findings in VR empathy research. And, I’m going to make the argument that physical movement in VR storytelling is itself an underused opportunity to create situations for empathy and generally change the way we think.

 

Blood and Chocolate: a variation on Queerskins: ARK at CannesXR/Tribeca Immersive

Queerskinsark_5

 

Blood and Chocolate a lyrical 360˚ video variation on Queerskins: ARK will be shown in the Museum of Other Realities June 24-26th, as part of The Cannes Film Festival with whom Tribeca Immersive is partnering. MOR is a virtual museum space, unfortunately only available for Oculus Rift and Vive headsets at this time.

This 360˚ video variation presents a glimpse of the intimate lovers’ pas-de-deux, choreographed by Brandon Powers to a score by composer Wilbert Roget II,  from Queerskins: ARK. That project – co-produced with Intel Studios and set for a 2020 release, harnesses volumetric video and spatial sound to tell the story of a mother who, reading the diary of her estranged son, Sebastian, whom she has lost to AIDS, allows herself to picture him alive and in love. The text used in the Blood and Chocolate installation and for the voice-overs in the experience are taken from Sebastian’s diary, available free in an interactive narrative at queerskins.com. For updates on Queerskins: ARK, please follow us @queerskins.

I will also be in conversation with Sarah Vick, Intel Studios, executive producer for Queerskins: ARK, who is giving a keynote on “Building a shared sense of Humanity with Volumetric Video” on June 25th.

The Importance of #Storytelling and #BLM

Part of my job as a physician is fact-finding, but the other part, the art of medicine, is storytelling. How I go about collecting a history, how I listen to and take seriously, and prioritize a patient’s concerns, and the trust or mistrust I engender, all affect how the patient receives the story of their illness and the subsequent outcomes. As a white woman working at Rikers Island Correctional Facility in New York City, I have had the opportunity to confront my own passive acceptance of a system which destroys black communities and functions as if inmates’ lives are disposable. And, as an artist/interactive storyteller, I have come to understand that how we as a country talk about race and injustice perpetuates the racism that is pervasive in our systems. Facts do not exist in a vacuum.  The way we tell stories matters. The New York Times publication of Tom Cotton’s call for military suppression of “insurrectionists” such as myself, a white woman doctor, mom, and artist, is one such glaring example of this.

The entrenched, pernicious narrative of the “good” and the “bad” mostly black protester/person is one that we see in the news as well in discussions with family and friends. The minute you fall into that narrative, you are supporting the racist system— not because acts of property damage and vandalism are not worthy of opprobrium or prosecution, but because you are privileging one kind of narrative over another much bigger and more important one. The narrative of “good” and “bad” overrides and is used to overlook the real violence done in so many ways to black people every day. The “bad” protester narrative is a distraction that plays into America’s underlying racism, whether acknowledged or not, that black people deserve less than justice because they are lesser, uncivilized, barbaric. This is a narrative handed down from slavery. It has infiltrated the subconscious of our white society so that we are not even aware of it. If you talk about vandalism, talk about systematic economic injustice, talk about systematic educational injustice. Give those equal time. This is not a white liberal justification for these acts of destruction. It is a plea for changing priorities in our storytelling. It was the exceptional brutality of George Floyd’s murder that lead to these protests. But, that exceptionality belies the fact that racism in American economic, political, educational, and social life is so pervasive and fundamental that for many people it is essentially a non-story—just the way things are. But, these are exactly the stories we need to tell, again and again, until we finally hear them and create substantive change. That story above all others is this: black lives matter.

Queerskins: a love story commercially released on the Oculus Store

We are live. We don’t usually ask that people share things for us on social. But, please share this information. Queer and otherwise non-conventional VR commercial release is rare. By supporting these kinds of experiences, you help insure a future for other artists.

In this Peabody Futures of Media award-winning virtual reality experience, you are seated in the back seat of an old Cadillac, behind grieving parents, Ed and Mary-Helen, as they take a magic realist journey down a country road in rural Missouri. Through the drama playing out in the front seat and by interacting with a box of personal belongings, photographs, and a diary, you get to know Sebastian, the son they have lost to AIDS.

Our goal was to create an experience that feels both comfortingly familiar and unsettlingly intimate and personal. In fact, many people relive their own childhood memories of sitting in the backseat, watching their parents fight, and being unable to leave or to intervene. Not only do you feel like you are really there, with the parents, this every-day act of sifting through the contents of the box connects you physically and psychologically to Sebastian’s life and the story. Also, because of your proximity to the parents, you read their body language through your own body, a little like sitting on stage with actors in a play. For this reason, we worked with a choreographer to orchestrate actor posture and movements. Ed and Mary-Helen essentially do a seated “dance,” a missed call and response, which reveals their conflict in a powerfully visceral way.  

In Queerskins: a love story, http://vr.queerskins.com/a-love-story, you become co-director of the story. Rather than offer a branching narrative or a set of defined tasks, we offer you an excess of possibilities for how to experience the story. You might get wrapped up in the parents’ fraught conversation as they struggle to come to terms with their son’s homosexuality and their Catholic faith. Or, you might spend part of the journey looking at the Missouri landscape, filmed on location. You might focus on examining the objects in the box. Depending on your own personal history and memories, you will relate and respond to the objects differently, and, thus, each person will construct the main character, Sebastian, differently. These choices as well as a randomization of the objects (there are three unique sets of photographs and seven diary entries that you might get each time) encourage replay of the experience, which enriches the story further.

Sound is a critical part of this experience. We were fortunate to work with Skywalker Sound for spatial sound design. Even before the visual story begins, as major credits roll, you are charged with constructing a story based upon what you hear around you: a screen door slamming, footsteps on gravity, a car engine starting. In this way, we immediately harness your  imagination, and you begin your role as co-creator.  Later, we use sound to augment the photographs. As you bring a photo towards you, it triggers an “aural memory” related to that photo. Lastly, you get to hear Sebastian’s voice reading his own diary. Recorded with binaural audio, he sounds as if he is right in front of you, a living presence, which you could almost reach out and touch.

Although this story is fictional, we purposefully incorporated elements of historical reality. All of Sebastian’s belongings are 3D scanned archival objects from the time period. The car is created with photogrammetry of a real 1986 Cadillac. The news on the radio is actual testimony before Congress in August 1990 at the beginning of the first Iraq War. If you are lucky enough to look out the window at a certain point, you will see a single billboard of an empty bed. This is a photograph of Cyril Tsiboulski’s bed, an homage to acclaimed artist Felix Gonzalez Torres who photographed his own bed after the loss of his lover to AIDS and placed it on billboards across the country. Through the appropriation of historically accurate content and technologies that capture the nuances of time-etched physical reality, we were able to create a magical realist aesthetic, which actualizes a tension between material, embodied reality and the quintessential human desire to transcend those limits through imagination and storytelling. Creating this unique aesthetic was not only an important artistic decision, it also reflects a key theme of the story. 

Queerskins: A Love Story  http://vr.queerskins.com/a-love-story is inspired and adapted from the critically acclaimed online multimedia narrative  Queerskins: a novel. http://online.queerskins.com/ that we put out in 2013, and which continues to be taught on a university level as an important work of digital literature. Designed as a multimedia scrapbook, it contains 40,000 words of Sebastian’s diary, two hours of audio monologues from five people who knew him, photos curated from Flickr Creative Commons, and crowd-sourced and commissioned videos. After experiencing Queerskins: a love story, you can further explore Sebastian’s life through this free online resource. 

Writer and director Illya Szilak, who is also a practicing physician at Rikers Island Correctional Facility in New York, was inspired to make Queerskins: a love story based on her experiences as a medical resident in NYC in the 1990’s. Howevery, she and Tsiboulski, who identifies as a gay man, did not approach Queerskins: a love story wanting to make an “AIDS story” or a “gay” story. In fact, these words are never uttered in the experience. Instead, through story and technology, the experience puts you in the position of living through the intimate, interior worlds of others that, hopefully leads you to an emotional engagement with the characters and themes, and, ultimately, an empathy for the characters’ unique personal experiences, and hopefully for all persons who experience love, illness and loss. Releasing Queerskins: a love story in the midst of a pandemic adds a powerful contemporary resonance. For many of us, during Covid-19  Ed and Mary-Helen’s grief is something we can immediately and personally relate to.  

Queerskins: a love story is available in English for Oculus Rift and Oculus Rift S (with Touch controllers) with optional Spanish or Italian subtitles. 

 

 

Queerskins: a love story Long-Awaited Commercial Release Next Week on Oculus Store

Queerskins: a love story, our Peabody Futures of Media award-winning Virtual Reality drama for Oculus Rift will be released on The Oculus Store next Thursday May 21st.

In this seated 6DOF experience, you are seated in the back seat of an old Cadillac, behind two grieving parents, Ed and Mary-Helen, as they take a magic realist journey down a country road in rural Missouri. Along the way, you will get to know Sebastian, the son they have lost to AIDS. “Queerskins: A Love Story” http://vr.queerskins.com/a-love-story is inspired by the online, interactive, multimedia narrative we put out in 2013 Queerskins: a novel  It tells the story of a young gay physician estranged from his Catholic family who dies in 1990, at the height of the AIDS epidemic. For the online, we created a multimedia scrapbook: 40,000 words of Sebastian’s diary, two hours of audio monologues from five people who knew him, photos curated from Flickr Creative Commons, and crowd-sourced and commissioned videos. Although you can navigate through any way you choose, you still feel more like a passive consumer than an active participant. Because Sebastian’s story deals with AIDS and homosexuality in a very intimate way, we were excited to tell it in VR so that we could make you a co-creator in the storytelling. In “Queerskins: A Love Story,” you are charged with reconstructing the man who has died by interacting with a box of his belongings, photos and a diary. How you relate and respond to the objects in the box, for instance, a statue of the Virgin Mary or homoerotic Tom of Finland drawing or a Hulk Halloween mask depends upon your own personal history. Many people relive their own childhood experience of being in the backseat, watching their parents fight, and being unable to leave or to intervene. Not only do you feel like you are really “there” in the backseat of the car behind his parents, this simple act of sifting through the contents of the box connects you physically and psychologically to Sebastian’s life and story. We were also aware that, because of your intimate proximity to the parents, (actors were shot with 3D volumetric video to capture the nuances of human movement), you will read their body language through your own body, a little like sitting on stage with actors in a play. For this reason, during rehearsals, we worked with a choreographer to orchestrate actor posture and movements. Ed and Mary-Helen essentially do a seated “dance”, a painful missed call and response, which reveals their conflict in a powerfully visceral way.

Although our story is fictional, we chose to incorporate elements of historical reality. All of the objects are 3D scanned archival objects from the time period. The car is an actual 1986 Cadillac. The news on the radio is an actual testimony before Congress in August 1990 at the beginning of the first Iraq War. If you are lucky enough to look out the window at a certain point, you will see a single billboard of an empty bed. This is a photograph of Cyril’s bed, an homage to artist Felix Gonzalez Torres who photographed his own bed after the loss of his lover to AIDS. Although, it would have been possible to 3D model these from scratch, we wanted to avoid the slick, hyperreal aesthetic that pervades much VR and digital film. Our goal was to create a magical realist aesthetic that generates a dynamic tension between embodied, material, historical reality and the very human desire to transcend those limits. To accomplish this, we ended up combining a lot of tech that no one has put into the same experience—360-degree video 3D volumetric live-action video, 3D scanned archival objects, photogrammetry of a real Cadillac, 3D CG modeling and animation, and spatial sound. Although the technical achievement of this project was recognized by SIGGRAPH, the Special Interest Group on Computer Graphics and Interactive Techniques, which showcased it in 2019, we are not interested in tech wizardry for its own sake, but for the potential it offers to tell emotionally impactful stories.

For us, sound is a critical part of creating immersion in VR. We were fortunate to work with Skywalker Sound for spatial sound design. Importantly, the experience begins with sound. As major credits roll, you are charged with interpreting the radio “play” you hear around you: a screen door slamming, footsteps on gravity, a car engine starting. From the beginning, your imagination is harnessed, and you begin your role as co-creator. Later, sound is used to create “aural memories” which emanate from the photographs in the box. Lastly, you will hear Sebastian’s voice, reading his own diary. Recorded with binaural audio, it sounds as if he is right in front of you, a living presence, which you could almost touch.

In this time of our current pandemic, Queerskins: a love story has acquired a contemporary significance which we did not anticipate. We do hope that it will encourage viewers to remember that prior epidemic, in which, unfortunately, some people were considered to have deserved their fate. For instance, Jerry Falwell, heard in an archival radio clip in the opening scene, preaching God’s love, also notoriously proclaimed that AIDS was God’s punishment for homosexuality. However, we did not approach Queerskins: a love story wanting to make an “AIDS story” or a “gay” story. We didn’t want it to be pigeonholed (or dismissed), or, even worse, create a story that suggests “gay people are just like straight people,” ignoring the political, social, and economic realities of being gay. Instead, through story and technology, the experience puts you in the position of living through the intimate, interior worlds of others. It is our hope that this will lead to an emotional engagement with the characters and themes, and, ultimately, an empathy for the characters’ personal experiences and, by extension, for all persons who grapple with love, illness, and loss.

Queerskins: ARK UPDATE

We are so pleased that ARK, the second episode in the Queerskins VR drama been chosen for The Tribeca Film Festival 2020. We hope to be able to share it with you soon.

Reading a diary left behind by her son, lost to AIDS, a devout Catholic mother living in rural Missouri finds away to transcend her self and her grief by imagining him alive and in love. Harnessing Intel’s 3D volumetric video technology, Wwise interactive audio software and spatial sound, Ark allows you to co-create the story through your body position and movements. In the first scene, you find yourself in a dimly lit attic. Here, we purposefully limit your agency with 360 video. A middle aged woman lumbers upstairs, carrying a cardboard box. She removes objects from it including the diary, which she begins to read. Entering her imagination, you suddenly gain freedom to move. How close you position yourself relative to the lovers determines how much of their intimate conversation you hear. With the last line, “touch me,” the two begin a dance choreographed by Brandon Powers (with beautiful score created for this piece by noted video game composer Wilbert Roget, II.) This transitions into a cathedral-like space. Here, your own body projects a colored light onto the dancers making them visible even in darkness. The light is most intense when your arms are abducted in a gesture of openness and vulnerability. The experience of the dance, choreographed in segments and randomized, is co-created by you. Depending on where you move in the space, you will trigger audio fragments of the mother reading the diary entry which inspired that segment of choreography. In one segment, if you move within a meter of the men, their movements slow and the sounds of their breathing override the score. At the end, the lovers fade. Turning towards the sound of pages turning, you see the mother reading the diary, and are gaze triggered back to the attic. You have lost your agency, but the room is different. Time has passed. The space is suffused with golden light and the sounds of a summer evening. Rising from her reverie, Mary-Helen catches sight of herself in a mirror. Touching her face and chest, she takes off her dress and contemplates herself in her slip. The slamming of a screen door makes her body tense, she reaches for her clothing, then decides against it and walks downstairs as she is.

 

Thoughts on the Coronavirus Pandemic

I share these thoughts. I  am, in fact, a practicing physician, board certified in infectious disease and internal medicine, as well as a writer and artist.

I am hoping that the experience of Coronavirus will not be limited to the blow by blow details, the endless talking heads on the T.V. news, but, will be also read as a symptom of a pervasive illness to which we as a human race are in danger of succumbing. Some of us will die from the virus, others will fall ill and recover, others will watch, asymptomatic, but, saddened and wearied by our inability to help. I hope that those who do survive will see the world and their place in it differently, just as a person recovering from a long, potentially fatal illness sees differently. We are all in this together. We are linked to the planet and each other and animals and insects and all creatures down to the bacteria in our guts. Science can help us figure out how to get back into balance, but, a willingness to take a really hard look at ourselves and a sincere willingness to make a better future for everyone (especially everyone “other”) is fundamental. This is a wake up call. We may not have another chance.

“Moments there were, when out of death, and the rebellion of the flesh, there came to thee, as thou tookest stock of thyself, a dream of love. Out of this universal feast of death, out of this extremity of fever, kindling the rain-washed evening sky to a fiery glow, may it be that Love one day shall mount?” last lines of Magic Mountain, Thomas Mann

Superman–a story from Rikers

I write vignettes of my experiences as a practicing physician at Rikers Island Correctional Facility in NYC  here.  This is the latest.

Superman
He tells me that he want to come off Suboxone, the drug that is being used to treat his craving for narcotics. He tells me that he hasn’t taken it for two days and he feels fine. It’s is an odd story. I am incredulous. His history is complicated, but it seems he got put on methadone upon arrest when his urine came up positive for opiates on a drug test. In jail, this just happens de facto. Very few people refuse. Even if they don’t actually need it, methadone numbs the pain. He tells me he has never actually used heroin. He was only taking pain pills for a gunshot wound incurred the month before. I tell him we don’t know why some people get hooked on narcotics and some walk away. I can walk away, he promises. Again, I try to suppress my disbelief. The story is not impossible, but it is far more likely that there is a motive of secondary gain. Housing? A program? He doesn’t want to be moved from this jail–that is clear, but, what he really wants, it seems, is just to be off the drugs. How his treatment was changed from methadone to Suboxone, is buried in the chart. “I quit methadone– 110mg– cold turkey,” he declares, with pride. Then, he tells me his story. But, it is not at all the story I expect. I stop poking at my computer as he tells it because, suddenly, an almost mythological image arises in my mind of a little boy squatting in the shadow of a massive chemical processing plant somewhere in rural Illinois. “I was out fishing with my father, we used to fish all the time. It was hot, so he told me to go back to the car and get some water. But, it was so far, so I went to the chemical plant and there was a basin by the side of the building and it looked like it was filled with water. Crystal clear water, so I scooped some up with my hands and I drank it, and I think it was water. But, ever since, I never get sick. ” We are both dancing around what he is saying. It’s the Spiderman story. He has given himself a superpower. “Mind over matter,” he says by way of explanation. He has the will so he will do it. I look into his light green eyes, his hair, graying, in long dreads, and I can picture him as a little boy. I imagine all children believe they are magical at some point in their lives. It is a response to the powerlessness of childhood. And, I do not find it surprising that a man in jail would return to that story and remember what that was like when life was new and grass green and sky blue and all the possibilities present on a Saturday afternoon, fishing with your dad. But, now, he is in jail. The story takes him back, and I see him as he was, and I do not take away that hope. I do not disabuse him of this wisp of a belief, that he does have a superpower, that he has been blessed or chosen, that his fate is not just this. I, too, want to believe it. I tell him I do believe there are people who can do superhuman things. “It is like Star Wars,” I say, “the Jedi mindset. If you practice it, you can do it.” He nods in agreement, “if you think you can, you can”. “Yes”, I say. I take his vital signs, and though it is not a miracle, his are normal. There are no signs of withdrawal. So, I take him off the pill

The End of Touch

manifesto

 

“Social distancing,” is a term we hear a lot recently. Although this phrase has emerged in the context of the Coronavirus, the fact is that we have already begun to practice it pervasively through our use of technology. The loss of life, economic value, productivity and the state of emergency, which for many of us is not new, but simply an amping up of the anxiety we’ve been feeling for the last three years, will come to naught if we do not see this as an opportunity.  Now is the time to seriously consider what is lost and gained as communication and “touch” become virtual.  I want to extend the term “touch” to encompass types of  existence in which having a vulnerable body is not something that is merely an inconvenient or horrible reality, one that must be dealt with, but an integral, valued aspect of living. In this way, I would extend “touch” to ecological systems. Our gut biome “touches” us. We “touch” our Earth, this only home for embodied creatures. To create and unlivable planet is to create a place without touch.

In 2007, Cyril and I put out an interactive narrative called Reconstructing Mayakovsky. (how I long to make this as a feature length VR!) It is 90,000 words of text –a proper novel–that is accessed through various “mechanisms”  including an “archive” based on internet links, that is necessarily, slowly accumulating 404 errors. I wrote it in response to 9/11. Inspired by Vladimir Mayakovsky, the Russian Futurist poet who killed himself in 1930 at the age of thirty-six, the novel imagines a post-nuclear war dystopia where uncertainty and tragedy have finally been eliminated through technology. It really asks what is lost and gained as the world becomes virtual. There is a very funny and very sad scene–you will have to play around to find it! in which the protagonist (VeraX) husband gets a blow job from a woman in “Real Time”, the toxic, almost unlivable world outside the virtual. There is also a fake investment video for the virtual world “OnewOrld” (so called because all difference, those inefficient drags on memory, have been erased–I will not go into the hegemony of what I call “Pixar aesthetics,” but you know what I mean) which decries the fact that “the human has become the weakest link in defense systems.” I did not make that line up, it’s a line taken straight from the DARPA website. You can access it here. The image above is from the manifesto I wrote. In the way of all good manifestos, it is over the top hyperbolic and poetic and aptly hysterical.  The design, created in collaboration with Pelin Kirca, replicates Roentgen’s original report of the discovery of X-rays in a German medical journal (accessed through that transcendent technology The WWW).  I love this image. X-rays make the body virtual, but Roentgen’s wife’s wedding ring, remains opaque, impenetrable.

The rise of big data and AI to provide answers to important questions, ranging from public policy decisions to whether your insurance will pay for a certain medical treatment, presumes that all that we need to know can be communicated into terms that disembodied beings can make sense of–the poetic, the intuitive, concepts like wonder and remorse are gone. (I  am a physician. I started out in molecular biology so I recognize both the incredible importance as well as the limits of science.) The emergence of emojis on FB at the same time that Oculus Rift became available to the consumer public is not serendipitous (nor is it an evil orchestration), it merely reflects the underlying understanding that there are aspects of human communication that need to be modified, commodified, codified so that machines can read them. But, to capitulate to the necessity of operating systems and RAM, is to capitulate to a loss of those aspects of being human -the nuances of voice, the tension in a neck, the half-smile, the body leaning toward you in conversation, which is part, is a fundamental (this is old brain) and incredibly pleasurable part of being human.

_ASG9105

The danger of touch has a history. Those who grew up in the 80’s when the transmission of a mysterious, deadly illness, now called AIDS, was still in question, remember this history.   Queerskins: ark, our newest VR experience,  revisits the story of Sebastian, a young gay physician estranged from his rural Catholic Missouri family who dies of AIDS in 1990.  In it, the mother reads his diary and allows herself to imagine him alive and in love. We filmed Michael DeBartolo and Chris Vo who play the lovers with Intel Studio’s 3D volumetric capture technology. There is very little dialogue in this. It was critical for us to tell the story through bodies and body movement because, on a meta-level, this work is about the dangers and pleasures of touch and embodiment. The last line of the dialogue is Sebastian’s who says “Touch me.” This segues into an intimate and at times very sexy pas-de-deux between the two men, beautifully choreographed by Brandon Powers.  The backdrop of the coronavirus puts the exhibition at risk, but also heightens awareness of the very question we are exploring.

I’d like to end these thoughts with a poem “Archaic Torso of Apollo”. I feel nowadays that only love and poetry are the only antidotes to this feeling of it being the end of the world. Rilke wrote this exquisite poem about seeing self in other and light as particle and wave. Art as physical and transcendent matter. Touching with the eye. The last line is utterly arresting. And, it means everything. So, yes, this can not be the new normal. We must change our lives. archaic Torso of Apollo

 

 

 

Final Rehearsal Queerskins: ark

In August we flew to Intel Studios in LA to film Queerskins: ark using their 3D volumetric  capture technology on a 30 ft stage –the largest in the world. Here are some beautiful photos by Asya Gorovits of Michael DeBartolo and Christopher Vo in final rehearsal in NYC.  We have made something that creates a kind of dialogue between your body and that of the virtual human characters. It’s intimate, wondrous–a new kind of “touch.” Coming soon!

_ASG9259_ASG9343_ASG9105