Nvidia’s last big talk was partially presented by a virtual clone of Jensen Huang, the company’s CEO. A publicity stunt for Nvidia’s tools which required a lot of technological investment.
Nvidia has just given new meaning to the virtual keynote. During the presentation of its Omniverse in April 2021, the company specializing in artificial intelligence and graphics card design deceived the audience by having part of the conference animate by a digital clone of Jensen Huang, the CEO of the ‘company.
Over the almost two-hour conference, 14 seconds in all were presented by a virtual avatar of Jensen Huang, almost without anyone noticing. These few seconds may seem small compared to the total duration of the conference, but they required a titanic work.
A clone created by AI
First, Nvidia teams created a 3D model of Jensen Huang by photographing it from all angles with a plethora of cameras. Next, an actor dressed in a motion capture jumpsuit re-read the CEO’s old speeches, mimicking his movements. This allowed Nvidia to create 21 models of 3D modeling which were all analyzed to create a compelling virtual skeleton. An algorithm has also been trained to replicate the CEO’s facial expressions to add a layer of realism.
Software called Audio2Face was then tasked with moving the clone’s face in 3D to match the text read by the real Jensen Huang. Another program called Audio2Gestures allowed Nvidia to animate the arms and body of the virtual mannequin according to what was read. By combining these two technologies, the 3D clone was able to express himself physically as a real human would.
The result of this experiment can be seen around 1:02:42 on the Nvidia presentation video. And if the virtual mannequin is a little stiff, the illusion still works quite well.
Towards more convincing deepfakes?
One of Jensen’s engineers in charge of virtualization explains in a documentary dedicated to the making of this conference that “ unlike the creation of a virtual character in a film, where it is possible to accumulate a lot of data, here we have very little content at the start “. The challenge was therefore to create a convincing duplicate from a 3D scan of Huang and not much else. The animation of the mouth, the synchronization of movements with the text, ” everything was done thanks to an AI », Specifies Nvidia.
The exploit here is therefore not to have created a convincing virtual human, video games have been doing this for a few years with great efficiency, but rather to create a virtual double from a very small data set. ” Our challenge was to create tools to create virtual humans much more easily. », Explains an engineer in charge of the project. The creation of a virtual Jensen Huang is therefore not just a simple publicity stunt: Nvidia’s algorithms could facilitate the creation of 3D characters, whether for video games or cinema.
One can imagine that by analyzing movie characters, their way of speaking and moving, it would then become possible to create convincing doubles. Perhaps even more convincing than what we have seen in recent Star Wars movies or series.