Alibaba wants you to compare its new AI video generator to OpenAI's Sora. Otherwise, why use it to make Sora's most famous creation belt out a Dua Lipa song?
On Tuesday, an organization called the "Institute for Intelligent Computing" within the Chinese e-commerce juggernaut Alibaba released a paper about an intriguing new AI video generator it has developed that's shockingly good at turning still images of faces into passable actors and charismatic singers. The system is called EMO, a fun backronym supposedly drawn from the words "Emotive Portrait Alive" (though, in that case, why is it not called "EPO"?).
EMO is a peek into a future where a system like Sora makes video worlds, and rather than being populated by attractive mute people just kinda looking at each other, the "actors" in these AI creations say stuff or even sing.
Alibaba put demo videos on GitHub to show off its new video-generating framework. These include a video of the Sora lady famous for walking around AI-generated Tokyo just after a rainstorm singing "Don't Start Now" by Dua Lipa and getting pretty funky with it.
The demos also reveal how EMO can, to cite one example, make Audrey Hepburn speak the audio from a viral clip of Riverdale's Lili Reinhart talking about how much she loves crying. In that clip, Hepburn's head maintains a rather soldier-like upright position, but her whole face not just her mouth really does seem to emote the words in the audio.
In contrast to this uncanny version of Hepburn, Reinhart in the original clip moves her head a whole lot, and she also emotes quite differently, so EMO doesn't seem to be a riff on the sort of AI face-swapping that went viral back in the mid-2010s and led to the rise of deepfakes in 2017.
Over the past few years, applications designed to generate facial animation from audio have cropped up, but they haven't been all that inspiring. For instance, the NVIDIA Omniverse software package touts an app with an audio-to-facial-animation framework called "Audio2Face" which relies on 3D animation for its outputs rather than simply generating photorealistic video like EMO.
Despite Audio2Face only being two years old, the EMO demo makes it look like an antique. In a video that purports to show off its ability to mimic emotions while talking, the 3D face it depicts looks more like a puppet in a facial expression mask, while EMO's characters seem to express the shades of complex emotion that come across in each audio clip.
It's worth noting at this point that, like with Sora, we're assessing this AI framework based on a demo provided by its creators, and we don't actually have our hands on a usable version that we can test. So it's tough to imagine that right out of the gate this piece of software can churn out such convincingly human facial performances based on audio without significant trial and error, or task-specific fine-tuning.
The characters in the demos mostly aren't expressing speech that calls for extreme emotions faces screwed up in rage, or melting down in tears, for instance so it remains to be seen how EMO would handle heavy emotion with audio alone as its guide. What's more, despite being made in China, it's depicted as a total polyglot, capable of picking up on the phonics of English and Korean, and making the faces form the appropriate phonemes with decent though far from perfect fidelity. So in other words, it would be nice to see what would happen if you put audio of a very angry person speaking a lesser-known language into EMO to see how well it performed.
Also fascinating are the little embellishments between phrases pursed lips or a downward glance that insert emotion into the pauses rather than just the times when the lips are moving. These are examples of how a real human face emotes, and it's tantalizing to see EMO get them so right, even in such a limited demo.
According to the paper, EMO's model relies on a large dataset of audio and video (once again: from where?) to give it the reference points necessary to emote so realistically. And its diffusion-based approach apparently doesn't involve an intermediate step in which 3D models do part of the work. A reference-attention mechanism and a separate audio-attention mechanism are paired by EMO's model to provide animated characters whose facial animations match what comes across in the audio while remaining true to the facial characteristics of the provided base image.
It's an impressive collection of demos, and after watching them it's impossible not to imagine what's coming next. But if you make your money as an actor, try not to imagine too hard, because things get pretty disturbing pretty quick.
Original post:
Alibaba's AI video generator just dunked on Sora by making the Sora lady sing - Mashable
- Shell to use new AI technology in deep sea oil exploration - Reuters [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Tom Hanks: I could appear in movies after death with AI technology - BBC [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Why C3.ai, Palantir, and Other AI Stocks Soared This Week - The Motley Fool [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- How to do the AI Webtoon filter going viral on TikTok - Dexerto [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- AI poses risk to humanity, according to majority of Americans in new poll - Ars Technica [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- New AI tool predicts Parkinson's disease with 96% accuracy -- 15 ... - Study Finds [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- AI is in a 'baby bubble.' Here's what could burst it. - Markets Insider [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Rise of the machines: how long before AI steals my job? - Mexico News Daily [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Amazon is focusing on using A.I. to get stuff delivered to you faster - CNBC [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Beijing calls on cloud providers to support AI firms - TechCrunch [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- AI at warp speed: disruption, innovation, and whats at stake - Economic Times [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- How a family is using AI to plan a trip around the world - Business Insider [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Prompt Injection: An AI-Targeted Attack - Hackaday [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- WHO calls for safe and ethical AI for health - World Health Organization [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Azeem on AI: Where Will the Jobs Come from After AI? - HBR.org Daily [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- AI runs amok in 1st trailer for director Gareth Edwards' 'The Creator ... - Space.com [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- From railroads to AI: Why new tech is often demonised - The Indian Express [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- How Generative AI Changes Organizational Culture - HBR.org Daily [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Google plans to use new A.I. models for ads and to help YouTube creators, sources say - CNBC [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- A.I. and sharing economy: UBER, DASH can boost profits investing ... - CNBC [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- A Wharton professor says AI is like an 'intern' who 'lies a little bit' to make their bosses happy - Yahoo Finance [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- CNET Published AI-Generated Stories. Then Its Staff Pushed Back - WIRED [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- AI-Driven Robots Have Started Changing Tires In The U.S. In Half The Time As Humans - CarScoops [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Elections in UK and US at risk from AI-driven disinformation, say experts - The Guardian [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Here's What AI Thinks an Illinoisan Looks Like And Apparently, Real Illinoisans Agree - NBC Chicago [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- We Put Google's New AI Writing Assistant to the Test - WIRED [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- 'Heart wrenching': AI expert details dangers of deepfakes and tools to detect manipulated content - Fox News [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- From Amazon to Wendy's, how 4 companies plan to incorporate AIand how you may interact with it - CNBC [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- Meta Made Its AI Tech Open-Source. Rivals Say Its a Risky Decision. - The New York Times [Last Updated On: May 21st, 2023] [Originally Added On: May 21st, 2023]
- For chemists, the AI revolution has yet to happen - Nature.com [Last Updated On: May 23rd, 2023] [Originally Added On: May 23rd, 2023]
- G7 calls for adoption of international technical standards for AI - Reuters [Last Updated On: May 23rd, 2023] [Originally Added On: May 23rd, 2023]
- Bloomsbury admits using AI-generated artwork for Sarah J Maas novel - The Guardian [Last Updated On: May 23rd, 2023] [Originally Added On: May 23rd, 2023]
- New AI research lets you click and drag images to manipulate them ... - The Verge [Last Updated On: May 23rd, 2023] [Originally Added On: May 23rd, 2023]
- France makes high-profile push to be the A.I. hub of Europe setting up challenge to U.S., China - CNBC [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- German tabloid Bild cuts 200 jobs and says some roles will be replaced by AI - The Guardian [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- How Christopher Nolan Learned to Stop Worrying and Love AI - WIRED [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- OpenAI plans app store for AI software, The Information reports - Reuters.com [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- A.I. could remove all human touchpoints in supply chains. Heres what that means - CNBC [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Cision Announces Code of Ethics for AI Development and Support ... - PR Newswire [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- AI Stock Price Prediction: Is C3.ai Really Worth $16? - InvestorPlace [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Is Applied Digital (APLD) Stock the Next Big AI Play? - InvestorPlace [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- 2 Cloud Stocks to Ride the AI Opportunity - The Motley Fool [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Digital health funding this week: Outbound AI, Aledade, Dexcare - Modern Healthcare [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- The AI Tool That Beat Out Top Wall Street Analysts - InvestorPlace [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Replacing news editors with AI is a worry for misinformation, bias ... - The Conversation [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- In new AI hype frenzy, tech is applying the label to everything now - Axios [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- How AI like ChatGPT could be used to spark a pandemic - Vox.com [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- 70% of Companies Will Use AI by 2030 -- These 2 Stocks Have a ... - The Motley Fool [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Why C3.ai Stock Crashed by 10% on Friday - The Motley Fool [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- YouTube integrates AI-powered dubbing tool - TechCrunch [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- AINsight: Now Everywhere, Can AI Improve Aviation Safety? - Aviation International News [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- What is 'ethical AI' and how can companies achieve it? - The Ohio State University News [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- US to launch working group on generative AI, address its risks - Reuters.com [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Amazon Wants to Teach Its Cloud Customers About AI, and It's Yet ... - The Motley Fool [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- HIMSSCast: When AI is involved in decision making, how does man ... - Healthcare IT News [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- How AI could transform the legal industry for the better - Marketplace [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Neuroscience, Artificial Intelligence, and Our Fears: A Journey of ... - Neuroscience News [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Why SoundHound AI Stock Was Making So Much Noise This Week - The Motley Fool [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Advertisers should beware being too creative with AI - Financial Times [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- 3 Top AI Stocks to Buy Right Now - The Motley Fool [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- 1 AI Stock That Could Take You to Easy Street -- and 1 That Could ... - The Motley Fool [Last Updated On: June 23rd, 2023] [Originally Added On: June 23rd, 2023]
- Generative AI To Wearable Plant Sensors: New Report Lists Top 10 Emerging Tech Of 2023 - NDTV [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- Researchers use AI to help save a woodpecker species in decline - MPR News [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- OceanGate fires a whistleblower, hackers threaten to leak Reddit data, and Marvel embraces AI art - TechCrunch [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- As AI Spreads, Experts Predict the Best and Worst Changes in ... - Pew Research Center [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- 9 AI-powered tools for empowering CFOs unveiled at Health Magazine round table - Gulf News [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- Translating Japanese, finding rap rhymes: How these young Toronto-area workers are using AI - Toronto Star [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- Artificial Intelligence in Asset Management Market to grow by USD 10,373.18 million from 2022 to 2027, Growing adoption of cloud-based artificial... [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- 5 Stocks Well-Positioned to Reap Rewards of AI: Morgan Stanley - Business Insider [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- ChatGPT-maker OpenAI planning to launch marketplace for AI applications - Business Today [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- AI watch: from Wimbledon to job losses in journalism - The Guardian [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- AWS is investing $100 million in generative A.I. center in race to keep up with Microsoft and Google - CNBC [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- Bets on A.I. and innovation help this tech-focused T. Rowe Price ... - CNBC [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- Generation AI: It is Indias time to play chief disruptor | Mint - Mint [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- The Next Token of Progress: 4 Unlocks on the Generative AI Horizon - Andreessen Horowitz [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- MongoDB Embraces AI & Reduces Developer Friction With New Features - Forbes [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- Why smart AI regulation is vital for innovation and US leadership - TechCrunch [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- WEDNESDAY: West Seattle facilitator hosting 'civic conversation ... - West Seattle Blog [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- A.I. has a discrimination problem. In banking, the consequences can be severe - CNBC [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]
- AI Consciousness: An Exploration of Possibility, Theoretical ... - Unite.AI [Last Updated On: June 26th, 2023] [Originally Added On: June 26th, 2023]