Prompt: Hollywood is going insane over an advanced new generative AI model called Sora that can create lifelike, movie trailer-quality videos in minutes from a few short lines of text.
This scenario became clear last week when OpenAI, the San Francisco-based technology company behind text generation app ChatGPT and image generation tool DALL-E, teased its latest project, the text-to-video AI model Sora. Ta. (The name is Japanese for sky, and was chosen by the creators to “evoke endless creative possibilities.” Or maybe they’re fans of Kingdom Hearts.)
The biggest person to sound the alarm after seeing what Sora was capable of was Tyler Perry. He told THR that he has put plans for an $800 million expansion of his Atlanta studio space on hold. “Jobs will be lost,” he said.
Sora’s video is impressive. A woolly mammoth marches towards you through the falling snow. People walk through the busy, snowy streets of Tokyo, and cameras swoop over buildings. “A gorgeously rendered papercraft world of a coral reef inhabited by colorful fish and sea creatures.” The final prompt was to create this impressive 20-second video clip of his created by OpenAI. specific prompts.
Filmmakers and developers within the AI community consider Sora to be a major advance and important step for generative AI, a tool that can already do things once thought to be far-fetched. Masu.
“That’s 18 months earlier than I thought it would be. It was a total surprise,” said Edward Saatchi, AI producer at Fable Studios. For him, the hype and excitement of seeing people create short clips and images with generative AI has “died down” and people are wondering, “When will he be able to see AI movies in theaters?” I started thinking about this more realistically. Sora, on the other hand, feels like a game changer.
“It’s getting a little overwhelming. Check out our latest 30 clips!” he said. “They were okay, but they weren’t at that level of sophistication.”
There had been rumors that OpenAI was working on video tools, but this announcement came as a shock. Even people on other teams at OpenAI didn’t know it was coming. While we’ve only seen what OpenAI has created using Sora, the tech company says the tool is used not only by some visual artists, designers, and filmmakers, but also by “red teams” looking for ways to exploit it. He said it was in the hands of. The public is not yet able to see the results or try out this tool in person.
Film director Paul Trillo, known for his acclaimed AI short story “Thank You For Not Answering,” is part of OpenAI’s test group. He received his Zoom demo of Sora on February 23rd. He was impressed by the quality of the video and its features, but until it becomes an open-source app that creators can fully customize and control, he wonders if it can disrupt the industry or if it’s just “great technology.” Corporate product demo.”
“There’s a long way to go from a single clip to creating tools that work in the form of a story that the viewer can watch without leaving the story,” he said. “I think it would be a great production for people who are still starting out in filmmaking and want to try and experiment with their ideas, but don’t have a lot of resources. But it’s all about control and what our true intentions are. I’m a little skeptical from a professional standpoint because it’s about how well you can execute on your vision.”
Sora is a step up from the models of startup Runway’s competitors and tech giants like Meta and Google. Sora’s high resolution masks the pixelated beauty of many generated videos and improves details such as skin texture, hair, reflections, water, and foliage. Sora also allows videos of up to 60 seconds. Previously, the limit was 3 to 8 seconds.
Saatchi said this is the biggest sign yet that AI movies are moving beyond two-minute shorts and closer to short films and TV episodes.
“We were reaching the limits of the kinds of stories we could tell in three- to eight-second shots,” Saatchi said. “We were in a rut as a community. This opens up the ability to tell more complex stories.”
Sora also has a good understanding of how things work in the world. Other generative AI video tools enable prompts to add staged movements and instructions that simulate camera movement. However, Saatchi said that Sora has unique background characters, realistic movement, and subjects that can interact and react. The videos released by OpenAI include images of waves crashing against a cliff, baby animals playing, and images reflected in the window of a moving train.
Trillo said he was also struck by what he called Sola’s “temporal coherence.” AI video can’t understand what’s happening in a shot from beginning to end. Extrapolate (or infer) motion from a single generated frame. Glitchy sequences occur, as well as “gumby legs.” Yes, there are videos of Sora where the woman’s legs switch mid-stride, but when it comes to walking, other models were walking so Sora could run.
Although the runways approached temporal coherence, Trillo said it was more of an “illusion.” OpenAI calls Sora a “world model” that operates in space and time, rather than a typical text-to-video generator. If generated video is “to be taken seriously, we need this level of consistency and control,” he says. “[Sora] I don’t think you’re guessing. I feel like the path is set. ”
Another thing Trillo cited as a big improvement (and a little “unnerving”) is Sora’s ability to break down prompts into time. The video shows a woodland creature flying around in the forest, and the clip ends with the creature encountering a mushroom with a fairy dancing above it. Sora understands the complex prompt sequence of events where multiple things are supposed to happen, bringing it “one step closer to becoming a usable storytelling tool.”
“We never had anything like that before,” he said. “That’s close to what I think you’re looking for.”
Other Sora assets include seamless video looping, which comes from its ability to understand motion, and “sampling flexibility” that allows you to view the same prompt from different perspectives, framing, or different aspect ratios.
There is also video-to-video editing where users can connect videos. OpenAI provided a demonstration of seamlessly integrating the two videos, showing a drone flying inside the Colosseum and a butterfly floating on a coral reef.
While most people were staring at the subject of Sora’s video, Trillo was transfixed by the background. AI often suffers from “occlusion” problems, where objects in the foreground change or disappear overtaking objects in the background. OpenAI said Sora still has some imperfections in this regard, but Trillo noted that in Sora’s video a person passes in front of the text on the wall and the text is consistent. I noticed. He said this shows that Sora is not just a diffusion-based model, but a hybrid of his more traditional 3D animation environments and special effects.
So should Hollywood be more afraid of being replaced by machines now than it was a few weeks ago, let alone six months ago?
“This is the first time I’ve ever felt like the ground was a little uneven or starting to give way, just like the illustrators felt a few years ago,” Trillo said. “I’m nervous, but I can’t help but be excited at the same time.”
Sora still has obvious flaws. First, there is no dialogue. According to Saatchi, the human mouth is one area that AI still doesn’t understand properly. The key is to make that happen. Sora can make one incredible 60-second shot, but that doesn’t translate into making a consistent movie.
“It looks great in a blog post, but let’s see how it works when you want to photograph the same person in the same location 10 times,” Trillo said.
Sora also looks a little too perfect. Trillo says it can lack the unpredictable hallucinations and imagination found in other of his AI tools. OpenAI is also very concerned about misuse of its tools, so it has strict parameters in place that prohibit applications related to sex or violence. (Filmmakers who tried to teach the AI that “this is ketchup, not blood” were disappointed.)
“We’ve got a new Hayes code,” Saatchi said. “You could make a very dramatic, theatrical movie, but that would be the worst thing for AI.”
An AI tool is only as good as its interface. Sora’s limited customization and clunky features prevent it from being adopted by filmmakers and at-home creators. Still, Trillo said these are “temporary hiccups” that could lead to large-scale adoption of Sora knockoffs.
“Maybe in two years, we’ll see an open source model that has a lot of control and gives filmmakers the level of detail they need,” Trillo said. “Easier and faster tools always win.”
Even if Hollywood wanted to use generated AI today, content created by AI cannot be copyrighted. Edward Clarice, a lawyer and managing director at Clarice Law Firm, said studios need to worry that anything they create will be protected and won’t be considered machine-generated by the Copyright Office. Told.
“Studios will have to be very careful not to incorporate generative AI into their processes,” he says. “They are essentially producing work that is in the public domain, so there is a big risk in incorporating generative AI into their workflows.”
Movies may not be ripe for destruction yet, but marketing certainly can. His 60 second clip of Sora is perfect for advertising. Trillo said the stock footage industry should also be concerned. Shutterstock recently partnered with OpenAI, and many of Sora’s models may be trained on OpenAI’s libraries. Trillo envisions a near future where Shutterstock will allow the service to create AI-generated videos for him instead of using existing stock footage.
Trillo believes that while some may allow Sola to falsely enter the industry, successful artists are those with traditional skill sets and vision. “My overly optimistic view is that people will continue to receive the same paycheck, but they won’t have to kill themselves for it,” he said.
Saatchi, who is part of the research team that developed an AI tool that can automatically generate episodes of “South Park,” believes we are inching closer to the world of automated showrunners. Content generated without real input from people can easily compete for attention with movies and TV.
“Is film a collaborative medium? That will be lost with fully automated content,” Saatchi said.
Still, he issued a warning. A year ago, AI advocates were ready to declare that everything had changed. It’s hard to imagine a role unaffected by AI, but so far “nothing has changed.”
“Every three years, Silicon Valley tells Hollywood that they’re going to completely tear it apart and change everything, and that’s it, and Hollywood is going to survive and thrive,” Saatchi said. “I don’t want people to worry too much about it, apart from the perspective that Valley is always trying to be disrespectful to Hollywood.”