Filmmaker Larry Engel on Video’s Augmented Future

Several years ago, Professor Kiho Kim from American University’s Department of Environmental Sciences and I (from the Film and Media Arts division) took a group of our students to Cuba, a truly interesting place, for research on the country’s water use and agricultural practices.

What a breath of fresh air, Kiho and I agreed, that there would be little internet service. The students would be off their phones and living in their world, the world around them. They would look out bus windows, they’d watch people on the streets, they’d notice the old cars, diverse architecture, beautiful buildings (some falling into deep disrepair) and meet Cubans of all sorts. 

No such luck. Even without the internet, the students spent much of their travel time on Snapchat. They took selfies and photos of each other but not enough of what was in front of them, the world of Cuba. Of course, they did interact with Cubans, attended lectures and visited many sites. They learned, explored, photographed and filmed urban farms, national parks and interviewed experts in agriculture, conservation and culture. They produced several good films out of their experience, so I shouldn’t be too critical.

But the trip emphasized for me, a documentary film maker since 1976, that this generation of digital and social media natives interacts with its environment in entirely different ways than previous generations. When I was a kid we went outside and played in the dirt and came in only when called for dinner. The members of this generation go outside but they do so with smartphones at the ready, with Pokemon Go driving them. This augmented reality is their norm, and it is becoming an increasingly powerful presence in video journalism as well.

Over the years, styles of visual storytelling and the technology that inform the narrative ─ the scaffolding of the story, if you will ─ have continuously evolved. During the 1960s, interview sound bites with politicians on the nightly news averaged around 40 seconds, according to one study. By 1988, it was down to about nine seconds. Pacing has dramatically changed too. Fast-paced editing is the norm. I think this is a result of shorter attention spans of the younger audience, and the competition for their eyes.

As technological advances led to miniaturized production equipment and associated cost reductions, there also was a democratization of film production. Distribution of visual stories was wrested from the hands of traditional broadcasters and then from cable outlets. Now with YouTube and other social media platforms such as Instagram, and with streaming services further competing for attention, the demand for new ways of storytelling and watching is strong. Netflix brought us binge watching, putting the entire season of a series out at once. This immersive experience is captivating to many.

And that is the key: Immersion. Our students immerse themselves in reality, and new augmented reality tools are making that immersion even stronger. Inevitably, innovation and technical advances by engineers and others have affected how we producers and creators of content tell our tales.

When cinema was in its infancy, its inventors were also the creators of the content. Frankly, if you watch any of the early Edison or Lumiere brothers “slices of life” or “films of actuality,” you would be shocked that audiences embraced them as powerful and impactful, let alone paid money to watch them. Documentary scholar and historian Eric Barnouw suggests that it wasn’t until artists and other creative explorers began replacing the inventors as film producers that the language of film ─ acting, fragmentation of reality into shots, adding twists and drama to their stories– truly began to evolve.

Today, there are other innovative technologies in their infancy that place the viewer directly into the scene of action ─ virtual reality (or VR), augmented reality (or AR), 360-degree video ─ and the trajectory is similar to those early days of cinema. The inventors are also the content creators in many cases, and we’re not sure yet what the “language” of this new media will become or what stories are best told. But it is clear that the two-dimensional world of cinema and video is no longer the only way to tell visual stories.

In AR, the viewer has control over where to look in that 360-degree space. One key element to AR is that early research suggests that this 360-degree world is more engaging than our current 2D world of film. Unlike 2D rectilinear filmmaking in which the filmmaker points to what the viewer sees by moving the camera or cutting to a specific shot of something, AR currently gives that control to the viewer. One question is how to get the viewer to look where the director/filmmaker wants them to look.

Currently, in most AR films, the viewer cannot move through that space. The viewer is in a fixed position. However, with some projects we add the ability to “transport” via a handheld device. This technique as well as actually moving the AR camera can lead to vertigo or nausea. It’s likely that these barriers to full immersion will be solved both technically and creatively within years.

There are several ways that we create augmented reality projects ranging from exclusively computer-generated games or experiences to spherical camera rigs containing six or more lenses to simpler two-lens systems. Cameras can cost tens of thousands of dollars down to under $500. AR films can be seen on phones or tablets and computers. You can change the field of view with a mouse or trackpad. Watching an AR film with goggles provides a more immersive experience in 360. (Oculus Rift is an example of a headset and earbuds that don’t use a smartphone, while Samsung makes the Gear VR headset for its phones. Other manufacturers include Google Cardboard, Zeiss VR and Merge VR, to name only a few.)

The world of AR seems to me a great way to immerse nonscientists in the scientific method, allowing the viewer to be something of a “fly on the wall” during experiments, research, excavations, and field work. It can engage the viewer in a way that is more powerful than watching a film.

Communicating science to the public has always been tricky, however. We must communicate fairly and responsibly about complex topics while engaging the audience with the life experiences of scientists, researchers, doctors, adventurers and others. I want viewers to share the excitement and passion my subjects have for their work. I also keep diversity and inclusivity in mind when choosing interview subjects. Such principles must guide our efforts in the new world of AR as well.

Several projects that I’ve viewed are indicative of the potential for augmented reality. “Why is Greenland Melting?” is a 5½ minute film about climate change and the loss of glaciers. Through a variety of capture techniques, we follow two researchers as they set up research stations, check data, and talk to the viewer about their work. Most scenes fade in and out to get from one location to another. The filmmakers also used the “transport” technology to allow the viewer to get closer to or farther away from the subjects or items. I was impressed with the AR experience that included flying in a research aircraft and in a helicopter. At one point, the viewer is actually outside the aircraft looking at a glacier from several thousand feet above. I must admit that my knees buckled and the person presenting the AR film apologized for not warning me what was coming.

While still in its infancy, 360-degree video is making strong inroads in journalism and science journalism. The New York Times releases a 360-degree story every day for its “The Daily 360” series. Here is one example, “The Land of Salt and Fire.” USA Today and The Washington Post are also producing 360-degree journalism. Most newspapers are becoming multi-modal distribution platforms, working hard to build out their websites, apps and social media to include both 2D and 360-degree reporting.

At American University’s School of Communication, where I am an associate professor, we have been teaching our students about the possibilities for these new video technologies. The university owns a 20-acre pasture in Warrenton, Va., that is being converted to a farm to produce vegetables for our campus food services. Trent Burns, an AU senior, did an independent study with me to create an AR experience that included projecting 2D rectilinear interview material on the 360-degree field. I am also working on two projects that focus on bringing 360-degree video of farms and forests to Washington, D.C. public school students,

Augmented reality also holds great promise for scientists as well as journalists. I teach workshops at American University and across the region on self-documentation of research, usually with smartphones and tablets. I believe that using inexpensive 360-degree cameras may be a significant new communication tool for scientists to connect their work and findings, and their passion, with the public.

# # #

Associated homepage image of 360-degree camera, courtesy of Larry Engel.