A group of Adobe researchers recently published a paper on a new generative AI model called VideoGigaGAN and we believe it may launch on a future product. What it does is upscale low-quality videos by up to eight times their original resolution without sacrificing stability or important aspects of the source material. Several demo clips can be found on the project’s website showing off its abilities. It can turn a blurry 128×128 pixel resolution video of a waterfall into footage running at a resolution of 1,024×1,024 pixels.
Post by @luokai
View on Threads
What’s noteworthy about the AI is it doesn’t skimp out on the finer details. Skin texture, wrinkles, strands of hair, and more are visible on the faces of human subjects. The other demos also feature a similar level of quality. You can better make out a swan swimming in a pond and the blossom on a tree thanks to this tech. It may seem bizarre to be focusing so much on skin wrinkles or feathers. However, it is this level of detail that companies like Adobe must nail down if they aim to implement image-enhancing AI on a wide scale.
Improving AI
You probably have a couple of questions about the platform’s latest project like how does it work? Well, it’s complicated.
The “GAN” in VideoGigaGAN stands for generative adversarial network, a type of AI capable of creating realistic images. Adobe’s version is specifically based on GigaGAN which specializes in upscaling generated content as well as real photos. The problem with this tech, as TheVerge points out, is that it can’t improve the quality of videos without having multiple problems crop up like weird artifacts. To solve this issue, Adobe researchers used a variety of techniques.
The research paper explains the whole process. You can read it yourself to get the full picture although it is dense material. Basically, they introduced a “flow-guided propagation module” to ensure consistency among a video’s frames, anti-aliasing to reduce artifacts, and a “high-frequency feature shuttle” to make up for sudden drops in detail. There is more to VideoGigaGAN than what we just described, but that’s the gist of it.
Potential inclusion
Will we see this on an upcoming Adobe product or roll out as a standalone app? Most likely – at least we think so.
In the past year, the company has been focusing heavily on implementing artificial intelligence into its software from the launch of Firefly to Acrobat’s new assistant. A few months ago during Adobe MAX 2023, a video upscaler referred to as Project Res Up was previewed at the event and its performance resembles what we see in the VideoGigaGAN demos. An old movie from the 1940s goes from running at a 480 x 360 image resolution to a crisp 1,280 x 960. Blurry footage of an elephant in a river becomes crystal clear. The presenter even mentions how the software can upscale a clip to four times the original quality.
Admittedly, this is conjecture, but it’s entirely possible VideoGigaGAN may be the engine behind Res-Up. Adobe’s future product could give people a way to upscale old family videos or low-quality footage into the movie we envision in our minds. Perhaps, the recent preview is a hint at an imminent release.
VideoGigaGAN is still deep in development so it’s unknown when or if it’ll come out. There are several obstacles in the way. The AI can’t properly process videos beyond 200 frames or render small objects, but we’ll definitely be keeping an eye on it.
In the meantime, check out TechRadar’s list of the best AI image upscalers for 2024.
