Big beauty brands using lip reading technology developed by U of T professor

ModiFace is now used by cosmetics companies from L’Oreal to Unilever and Yves Rocher for different applications, such as adding different shades of lipstick to an image of a face uploaded by a user. (Screengrab from Modiface’s website)

TORONTO – Canadian technology originally developed years ago to enhance speech recognition is now being used by some of the world’s biggest beauty brands.

University of Toronto engineering professor Parham Aarabi designed lip-reading software more than a decade ago that initially garnered interest from the military community.

“The goal was to read lips and combine that with audio to do better speech recognition,” Aarabi said.

“The idea was that in certain environments – noisy environments or very far away from the person you’re trying to listen to – audio doesn’t work, so you have to use computer vision to read someone’s lips to guess what they might be saying.”

The technology worked about half the time, he said.

“You could eavesdrop on someone from very far away and based on the image zoomed in on their face guess as to what they were saying.”

But the algorithms he built to track contours of lips proved adept in the beauty industry.

That’s when his company, ModiFace, was born. The company grew slowly since launching in 2007.

Aarabi says the technology is now used by cosmetics companies from L’Oreal to Unilever and Yves Rocher for different applications, such as adding different shades of lipstick to an image of a face uploaded by a user.

About a year ago demand for ModiFace began to surge. At the time, he said, there were about 20 brands using the software.

Now, he said, 52 brands have licensed his technology that appears in more than 200 apps, with a total of more than 60 million downloads.

“I don’t know what the catalyst was, but everyone is rushing to have augmented reality as part of their apps, websites and retail solutions,” Aarabi said.

He expects that total to double in the coming year.

It turns out his lip-reading software solved a long-standing problem.

“One of the biggest challenges in the beauty industry is that most people don’t know what product is best for them,” he said.

“The way the beauty industry has worked for a century is you’re shown what a model looks like and because it looks good on a model – or because the skin care makes the model look perfect by some definition of perfection – therefore this product must be the right product for you,” he said.

“What we found is that if we can show the product in a very realistic and truthful way to the consumer on their own image… it is substantially more effective as a marketing and educational tool than if you show an image of a model.”

In the next year or two Aarabi says his technology will allow customers to walk into a store, glance at a mirror and have different shades of lipstick on their reflection looking back at them.