The future of mocap in VFX and virtual production

With the explosion in virtual production over the past years, motion capture has been becoming even more important for productions across the industry, including high profile uses in the likes of The Mandalorian. 

Vicon is at the forefront of this sector, providing its cameras for use in life sciences, virtual reality, and engineering, as well as in media and entertainment. It recently released a new flagship range, Valkyrie, that includes a 26-megapixel model and a high-speed capture mode at up to 2,000 (FPS) when windowing techniques are uses (500FPS at native speeds).

Other upcoming releases include the SuperNova system for tracking of props and objects, as well as its Crown Camera Tracker for use in virtual production.

Chief technology officer Mark Finch told Broadcast Tech that the latter brings its own challenges, even when compared to when Vicon’s services are used for medical decisions or infrastructure that thousands will use.

Finch admitted: “They’re willing to try the craziest stuff. They’re the most likely to adopt something that’s not quite there.

As a result the media industry gets first look at a lot of the tech: “VFX are generally early adopters, and then clinical uses will come later. That’s roughly the pathway of where we push us innovation and get early feedback in order to have a true and tested solution.”

A growing part of this demand for the latest tech comes from virtual production – for example Vicon’s work on Disney+’s The Mandalorian saw the need to film the walk into an LED Volume stage – and Jeffrey Ovadya, sales director for the Americas at Vicon, gave some insight to this boom: “I think [virtual production] is going to get a lot of attention, but it’s a very narrow field that needs extremely high precision and accuracy. That’s why when you hear about it, it’s because they bought 60 cameras. That’s a reason why it’s at the forefront of the business, but it’s just one piece of the puzzle.

He added“I was just talking with the head of virtual production from Netflix [Girish Balakrishnan], and I asked, ‘how are you managing all of these stages and all of these different productions?’ It’s really education and best practices.

“So many people are now trying to enter this, and it’s their first foray into virtual production and they’re like, we’ve got a lot of money, we’ve got a studio, let’s go go go. They forget to think about how they are going to actually create a pipeline that’s efficient and spit out the quality needed.”

Even with that the case, traditional VFX is still an important area for mocap: “So yes, it’s going to be the biggest thing in the news, but VFX is still a big part of performance capture. It’s a big part of these VR volumes. When you think about Ready Player One in 2016, that was one of the first films that filmed in virtual reality with the players in their headsets while being mocapped.”

When it comes to what’s up next for mocap and the media industry, first is likely to be the, possibly less exciting, issue of reliability. Finch said: “It’s giving more robustness around that tracking, which is the difference going forward. Reliability is essential for these production stages, because if anything breaks production shuts down and things will get thrown out.”

However, after that, there are a number of emerging technologies that could have an impact on the sector. Finch enthused: “The introduction of machine learning into motion capture is a very exciting space.

“The science is at a point now where it’s true, tested, and showing enough promise that we have been thinking seriously about how that can be integrated into solutions, whether it be marker tracking - improving the tracking of your markers using a machine, to the other extreme of markerless. There are boundless opportunities for embedding these scientific innovations, which are existing solutions, to reduce blocks for the customer.”

However, according to Ovadya: “Outdoor markerless systems have a way to go. We’ve looked at the data from some of them, and it’s messy.”

He added that there are a number of other technologies that could change the game: “We’ll see if quantum computing enters the fold, or distributed processes, AI.”

Finch said that the most interesting breakthroughs may come more from the user-side than companies such as Vicon: “That’s the exciting thing, we just build it and then they’re the ones that actually use it. Everything we’ve put in the camera has come from customer requests and trying to keep pushing the boundary of innovation, with the amount of data, the number of megapixels, etc. that you can fit into this small package. Now, let’s see what the industry does.”