Procter & Gamble ‘Reads’ Facial Expressions of Shoppers
And Then Adjusts Ad Displays

By Dale Buss 

CPG brands, retailers and technology companies are experimenting with all sorts of ways to harness sensors and data analytics to make shopping more interactive and promotional, prompting real-time decisions at the shelf that benefit all the business parties as well as consumers.

Among the most interesting tests is something that Procter & Gamble is doing with a tech startup that “reads” shoppers’ facial expressions and then adjusts the digital information and advertising displays in front of them based on whether they’re paying attention to the marketing message of the moment, and on whether they’re demonstrating emotions such as delight or displeasure through their facial cues.

Developed by San Diego-based Cloverleaf, shelfPoint is a strip of digital display that wraps around retail shelves and is equipped with optical sensors and powered by an artificial-intelligence algorithm. ShelfPoint can pick up anonymous information about shoppers such as age and gender as well as instantaneously amend marketing messages based on readings of 40 different facial attributes ranging from liftings of the eyebrows to wrinkling of the lips.

Impulse buys and purchases of Procter & Gamble brands and other products were nearly 40 percent higher with shelfPoint than with the average store shelf or with cardboard promotional material, according to Cloverleaf. The company also is testing shelfPoint with Dell Computers.

“Think about the possibilities for the health and beauty area with P&G brands, for example,” Gordon Davidson, founder and CEO of Cloverleaf, told CPGmatters. “You may have a message for Oil of Olay that’s designed to grab your attention, and then you turn your head and we’re educating and presenting on the cleansing product. Then you’re invited to the second step, maybe a moisturizer, and then some third step.

“So the idea is that each of these Oil of Olay products is presented on the shelf, but it’s also a regimen sale, and you’re educating people using shelfPoint about why they would want to pursue a regimen.”

Another tested prospect using P&G brands and products involved multiple brands in a single promotion, such as Tide detergent and Bounce softener and even a third, non-P&G product such as a paper-towel brand. “P&G could have a ‘better together’ message designed to draw your attention, and then once it does, there’s separate messaging for Tide and Bounce,” he said.

The tests were conducted at P&G's Beckett Ridge Innovation Center in Ohio.

Another twist on the P&G test has had the company using shelfPoint to simulate promotion of Tide with Bounce in one group of stores and Tide with some other messaging in a second group of stores, which provides instant comparisons in the effectiveness of the varying messages.

“Right away you can see which message is resonating, and you can see the demographic in front of the shelf and their emotions,” Davidson said. “If you don’t see a lot of joy and happiness but instead frowns and contempt, maybe the messaging is off.”

Procter & Gamble declined to make an executive available for an interview about its shelfPoint test.

ShelfPoint is a combination of optical sensors, software and shelf hardware that consists of digital displays that come in units about two inches tall by about two feet long. The displays can be combined as components of a larger image, and the movements take into account the combined “canvas.” Or shelfPoint displays can be arrayed “venetian blind” style, Davidson said, so shoppers can see both shelf content and the live messaging.

The system’s sensors observe and absorb information about seven different emotions that shoppers express based in facial attributes, he explained. And while Davidson conceded that few consumers react with wide-eyed, outright joy, or frown-producing chagrin, when confronted with a supermarket display, he maintained that even minimal changes in expression typically come over the faces of shoppers when they see something new.  shelfPoint can register and interpret a slight smirk or uplifted eyebrow.

“People are more expressive than they think they are in these situations,” he said. “And sometimes people will tell you what they think you want to hear. But their face doesn’t lie.”

While it isn’t yet clear what P&G or other CPG and consumer-goods companies might do to harness shelfPoint, the next phase of testing the platform, Davidson said, is to use “nested content so you can come up with mass customization.”

In other words, brands could use their data about demographic preferences or dislikes for their brands and products. When shelfPoint recognizes a shopper of an apparent particular demographic type, the program would respond appropriately. “But there are no privacy issues with shelfPoint, Davidson noted, because the program isn’t a facial-recognition platform. All data points generated are anonymous.

Among other things, Cloverleaf is recommending to brands and retailers participating in the tests that they create areas in the stores where shoppers can rely on seeing shelfPoint messaging continually. “That is subtly training your customers about where are the new and special things in the store, and because they are dynamic and digital, those places don’t get stale,” Davidson said. “What does get rearranged is the shelf layout.”

Other tests of “smart shelves” in the industry range from Kroger’s work to detect individual shoppers through their mobile devices as they walk through the store, which produces offers of tailored pricing on specific items, to test by King’s Hawaiian breads and retailers of shelves that generate data on out-of-stocks.

Davidson said that P&G and many other CPG companies are eager to probe shelfPoint’s capabilities because it provides point-of-sale information that can be used in real time instead of after the sale.

“The more robust information that we can provide,” he said, “companies like P&G say give me as much as you can, because it’s information they don’t have today.”

                                                                        Early February 2017