Barry Sandrew on bringing colour to classic films, 3D to new feature films and the potential of VR/AR

Barry Sandrew is a neuroscientist who invented a process for colorising B&W films and bringing 3D to 2D movies! He was kind enough to have a Q&A on his experiences and what he sees in the future for VR/AR. He’ll be speaking about the journey of alternative realities at Magnify World in Melbourne this month.

Barry Sandrew, Ph.D. and a few of the titles he has worked on via colorization, 3D and VFX

1. How has a neuroscientist background affected how you look at technology overall?

I got into feature film production when a group of investors enticed me to leave my position as Staff Neuroscientist at Harvard Medical School and Mass General Hospital to invent colorization. At the time I was conducting research focused on improving the diagnostic value of digital medical imaging using machine vision (the precursor of AI), pseudo-color and voxel based 3D reconstruction of CT, MRI and PET. The investors recognized that my knowledge and experience in digital medical imaging as well as my understanding of the human visual system was a perfect combination to solve the problem of digital colorization. This was a time when there was a renewed popular interest in iconic vintage movies, creating the perfect environment or zeitgeist for inventing colorization. From a business perspective, the incentive was clear. If you colorize a black and white movie that has fallen into public domain, you receive a new, 95 year copyright on the colorized version as a derivative creative work. In essence, anyone who could get their hands on the most iconic public domain films ever produced could own them… or at least they could own the colorized versions. The black and white versions would remain in public domain. To make a very long story short, the colorization process that I ultimately invented provided a creative way of visually enhancing those magnificent character driven movies, making them more accessible and acceptable to a wider and younger audience. For me it provided a unique and potentially lucrative opportunity to leave academia and become an entrepreneur in the feature film industry and I took it. Colorization was a great ride!

In 2007, when I began inventing the process of converting 2D movies to 3D, my neuroscience background became even more of an asset. It was the foundation for building technology that effectively removes the fourth wall of the theater for the movie going audience, making feature films truly immersive. Digital 3D cinema made it possible to transform the flat movie screen into a storytelling window where the audience could sense infinite depth within the screen as well as physically experience the story throughout the theater. For the first time it was possible to engage and/or infringe on the personal space of each member of the audience, recruiting the most primitive parts of the brain to enhance the story telling process.

I’m now exploiting my neuroscience background in the design and creation of Augmented and Mixed reality experiences. In Mixed Reality we have the potential of intimately melding computer generated experiences with our sensory receptors in a manner never before achievable. Once the technology has matured, I believe we will be able to creatively synch our neuro-cognitive reality with virtual reality in a way that will make it difficult if not impossible to determine where our personal reality ends and artificial, alternative realities begin. At that point we’ll have achieved the ultimate immersive state in storytelling.

2. Give us a bit of info on what you’ll be talking about at Magnify World, what’s the focus of your talk?

The focus of my participation in Magnify World is: The journey of Alternative realities… where they’re headed and how they’ll influence our lives in the future.

Over the past two years the scope and subject matter of my Magnify World presentations have not changed considerably. Last year I presented my take on the current status of alternative realities and pointed to predictions I’ve made over the past three years that came to pass, along with compelling examples. This year however, my focus is on the next 18 months which I consider pivotal for Augmented Reality and Mixed Reality. I believe my perspective, based on decades of innovating within the feature film and visual effects industries combined with my extensive networking within the VR/AR/MR community will be both informative and entertaining to the creative, production and investment communities as well to the general consumer.

3. As someone who invented digital colorization and a process for converting 2D films to 3D — do you see this move to virtual and augmented reality as the same sort of shift? How will the way we consume content change as a result?

If you look at my LinkedIn posts and listen to my podcasts (The AR Show Part 1, The AR Show Part 2) you’ll clearly understand that I learned a great deal from my Hollywood career in visual effects, particularly as it relates to colorization of black and white movies and the 3D conversion of 2D films. That education gave me a unique perspective on the consumer mindset and its level of acceptance of new technology, at least as it relates to the United States.

The big question is… which immersive technology or combination of immersive technologies will ultimately resonate with the mass consumer and become ubiquitous. You can create the most awesome alternate reality experiences but if the technology required to deliver them is not widely accepted by the average consumer and consequently cannot generate sufficient revenue then it becomes an art project or a hobby.

While pontificating over the design of electronic consumer devices, Steve Jobs declared, “You have to start with the consumer and work backward to the technology, not the other way around.” I believe that Hollywood became so enamored with Virtual Reality that it created an echo chamber of enthusiasm that was void of any real consideration of whether the consumer would ultimately accept it. Virtual Reality has not yet attracted a mass market that can generate revenue outside of site based venues, eSports, gaming and a number of enterprise apps. If you ask VR practitioners and evangelists, they’ll tell you that fully immersive Virtual Reality is a work in progress. One has to be patient as they “wait for the technology to evolve and mature.”

While we’re waiting for fully immersive VR to mature. I believe that innovators in Mixed Reality, a medium that has evolved out of utility based Augmented Reality plus Virtual Reality have more effectively considered the type of use cases that will ultimately resonate with the consumer. They are now focusing on consumer tolerance and acceptance toward smart glasses similar to our smart phones which have already become extensions of ourselves. Over the next 18 to 24 months, AR and particularly MR will start to capture the consumer consciousness and begin to build a lucrative industry.

4. Was there any backlash to your work with colorization and 2D/3D conversion? Are there lessons to be learnt from that when it comes to moving towards VR/AR?

The Saga of Colorization

Oh yeah… The backstory of the backlash on colorization is a fascinating book in its own right and I’m so happy to have been a central part of that story. I believe that one of the biggest issues with colorization was that the profits generated from the technology were going into the pockets of my investors rather than to the directors. Most of the directors and producers couldn’t care less about the creative aspect. Indeed, if any of the opponents of colorization truly understood the limitations of the technology they would have to admit it was much ado about nothing… The bottom line: The more that Woody Allen, Jimmy Stuart, John Huston, Anjelica Huston, Frank Capra, Sonny Bono and the rest complained… and the more U.S. congressional hearings that were held on the topic, the more money that rolled into colorization. There was a significant consumer demographic that wanted it and demanded it. Why else would I have continued to produce it?

In reality, colorization actually subsidized the restoration of vintage black and white prints. In order for my teams to colorize a feature film, the black and white version first had to be restored and often stabilized. Prior to colorization, there was absolutely no monetary incentive for anyone to restore public domain movies because there were hundreds of copies out there for purchase as VHS tapes, CD and DVDs and just about all were in horrible condition. Hackers just threw out multigenerational copies just to make a buck. My colorization process provided the financial incentive to restore those classics. On top of that we always included the pristine restored black and white version alongside the colorized version in every DVD release, so the consumer could decide which they wanted to watch. I was villainized for destroying America’s film heritage when actually I was preserving that heritage in its original black and white format. Few are aware of this.

But regardless, the only people that really mattered to me were the consumers. There were two audience camps in colorization. There was a group that was fervently if not cultishly in favor of colorization and wanted more than we could produce, but there were also some that thought that colorization got in the way. These classic films were great character driven stories that were almost like stage plays only cinematographically staged by incredible artisans of the craft. There was a part of the audience that simply wanted to get lost in the story and did not care to be distracted by color… they looked upon classic films like a good book. Some of the perennial films, like It’s a Wonderful Life were watched by entire families at around the same time every year during Thanksgiving and Christmas…. always in black and white. It became a family tradition.

To the younger generation, colorization made the film real… it became a story to which they could relate in color rather than simply tolerate in black and white. Indeed, color brought out visual detail that most people never knew existed. To most adults that detail was a fascination that enhanced the classic film, to others it distracted from the story. Those who considered it a distraction were generally people who saw the film dozens of times over the years and could actually recite the dialog by heart. To them colorization became a new and unique experience that they did not expect nor want. It changed the story for them in a way that was not positive. And of course there was also a population that couldn’t care less either way.

Today, in a world where Ang Lee’s Life of Pi, in reality a computer animated film can win an Academy Award for Best Cinematography and where most movies are shot against green screen rather than on location, where practical (or real) special effects are more often than not replaced by computer generated visual effects… a new movie going generation has grown up that accepts if not expects creative enhancement of feature films via computer imagery. Colorization is now less of an anomaly than it was in the past. Indeed, I believe that the colorized versions of It’s A Wonderful Life, Holiday Inn and Miracle on 34th Street that I produced between 2006 and 2009 for Paramount, Universal and Fox respectively were the highest performing home entertainment releases in the history of those films.

3D Movies

With 3D, the problem was that:

1. Some people didn’t want to wear glasses because it separated them from their environment and believe it or not… the screen
2. Some people felt the glasses made them look foolish.
3. There were others who, aside from the glasses issue, did not want to pay an upcharge to see a movie that they would be fine watching in 2D.
4. And then there were the people who, like with colorization… simply wanted to be told a story and did not want their personal space invaded. They wanted to enjoy the movie without being personally immersed in the movie, even passively.

From a neuroscience perspective, 3D conversion can be designed to recruit the more primitive parts of the brain that include the amygdala and hippocampus of the limbic system. Often when an object in a 3D feature film comes out of the screen and enters the personal space of an audience member, that visual effect activates flight & fight brain mechanisms within the limbic system that dramatically increases the emotional and adrenal effect. While some audiences love that heightened sense of immersion, I believe there is a significant population that just wants to experience the story that, if well crafted, should be sufficiently emotionally charged without engaging the audience’s physiological survival instincts.

So how does this relate to Virtual Reality and both Augmented Reality and Mixed Reality?

I don’t think it takes a brain scientist to see the link and the lessons learned from colorization and 3D movies.

Virtual reality not only puts the audience smack dab inside the story but in its purist interactive form, it also puts the burden of story creation on the audience. In effect, it takes the storytelling out of the story and makes it a personal experience. It also totally alienates the audience member from the rest of the world and becomes an isolation experience except perhaps for avatars which brings up the whole issue of tacky CGI avatars versus “Uncanny Valley” avatars which is another topic of discussion.

In Virtual Reality we’ll find the same three types audience responses that we saw in colorization and 3D but to a greater degree of differentiation. There are those who absolutely love to be totally immersed in an experience that some, other than me would call a ‘story.” Then there are others who find the technology and experience fascinating in small bites but for entertainment value, they consider it uncomfortable, unnecessary and missing the story-telling mark. I contend the later is the larger demographic. I don’t think there will be a sector of the audience that would be neutral about the medium, though I could be wrong.

6. There is still a large level of uncertainty around viewing stories in VR — people don’t really look around much when they’re in VR and it has seemed like quite a challenge to guide the viewer’s gaze compared to traditional mediums. What are your thoughts on this?

Indeed, it’s very expensive to create a 360 degree stereoscopic world of which most is irrelevant to the story. I believe that in the past free ranging Virtual Reality was simply technology for technology sake. This is where the definition of creative storytelling and directing come in. There should be no need for the extraneous parts of the world to exist if the story and visuals are sufficiently engaging. When Ang Lee was asked what he thought of virtual reality, he said… “my Virtual Reality is better than your Virtual Reality. Let ME move the camera.” Personally, I would much rather become engaged if not emotionally immersed by a story that Ang Lee is visually presenting to me rather than be fully immersed in an experience that I’m expected or allowed to create myself.

7. Was there anything in particular you did early on in your career that really helped your success?

My science background definitely gave me the ability to think critically but of equal importance, it gave me the ability to view intellectual problems from a global perspective and turn those problems into innovative opportunities… recognizing and gathering elements that create a whole greater than the sum of its parts that can be used create a successful product or endeavor. I think that’s also called being a successful entrepreneur.

8. What advice would you give to developers and tech enthusiasts who want to make a change in the industry?

Do not follow the crowd… think for yourself… follow your own dreams and not those of others. But most of all, always do a reality check to make sure you are not in someone else’s echo chamber that has little chance of succeeding.

9. Anything I haven’t covered which you’d like Magnify World attendees to know or think about?

I would like to point out a very cool feature of this year’s Magnify World. This year we will be introducing a unique Augmented Reality experience for all of our attendees. It’s called Social-Recall… a face sharing Augmented Reality app for iOS and Android that allows everyone at Magnify World to recognize any other attendee via their live smart phone camera. I consider it a perfect example of what will become an invaluable productivity app for professionals that also promises to be a great deal of fun as a social stimulant.

This is the way it works: Everyone who is attending Magnify World will have the option of registering within the Magnify World Social-Recall app. Within that app, each attendee will be prompted to take two selfies and type in both their name and company affiliation. The app will then automatically link to their LinkedIn page if one exists. This information is then stored in the cloud along with the identical information of all other attendees who similarly registers.

If all of the attendees register through the Magnify World Social-Recall app there will literally be no strangers at the event. When you’re at the exhibit area, at any panel or at the VIP cocktail party you’ll be able to scan the room and recognize virtually everyone along with their company affiliation and you can see their LinkedIn profile… all in Augmented Reality. Incorporating the most advanced facial recognition available today from Imagus Technology and Bident Solutions, Social-Recall becomes a super business card hanging over each attendee’s head. Like the popular 80’s television show, Cheers… wherever you go at Magnify World, “everybody knows your name.” I can’t imagine a better opportunity for promoting and stimulating social interaction than that. Here’s the registration URL: https://conferences.imagus.com.au/magnify_world

A huge thank you to Barry Sandrew for taking the time to share his story, his advice and his view on where the industry is headed! If you’d like to hear more from him, he’ll be at Magnify World in Melbourne on the 24th August.