New Amazon Just Walk Out multi-modal foundation model represents latest innovation in checkout-free stores

Amazon has announced an advancement in the technology behind Just Walk Out, its checkout-free service for retailers.

A new multi-modal foundation model increases its accuracy by using the same transformer-based machine learning models underlying many generative AI applications, and applies them to physical stores.

In an online post, Jon Jenkins, Vice President, Just Walk Out technology, AWS Applications, said: “We accomplish this by analysing data from cameras and sensors throughout the store simultaneously, instead of looking at which items shoppers pick up and put back in a linear sequence.”

“For retailers, the new AI system makes Just Walk Out faster, easier to deploy, and more efficient. For shoppers, this means worry free shopping at even more third-party checkout-free stores worldwide.”

Just Walk Out uses cameras, weight sensors, and a combination of AI technologies to enable shoppers in physical stores to buy food, beverages, merchandise, and more, without having to wait in a checkout line, or stop at a cashier.

The technology, which launched in 2018, was built using genie and machine learning available at the time to figure out “who took what.”

Previously, the AI system analysed shopper behaviour sequentially - their movement and location in the store, what they picked up, and the quantity of each item - each action processed one after another.

However, in unusual or novel shopping scenarios (such as if a camera view was obscured due to bad lighting or a nearby shopper), the sequential approach could take time to determine purchases with confidence, and sometimes required manual retraining of the model.

Jenkins said: “The new Just Walk Out multi-modal foundation model for physical stores is a significant advancement in the evolution of checkout-free shopping.”

“It increases the accuracy of Just Walk Out technology even in complex shopping scenarios with variables such as camera obstructions, lighting conditions, and the behaviour of other shoppers, while allowing us to simplify the system.”

“The new Just Walk Out AI system is able to achieve higher levels of accuracy by analysing all sensor data simultaneously, rather than sequentially. It looks at multiple inputs - cameras, weight sensors, and other data - and prioritises what’s most important to accurately determine the variety and quantity of items selected.”

“It also uses continuous self-learning and transformer technology, a type of neural network architecture that transforms inputs (sensor data, in the case of Just Walk Out) into outputs (receipts for checkout-free shopping).”

For example, a shopper might pick up and put down multiple varieties of yogurt, in different combinations, and as they are doing so, another customer might reach for the same item, or the freezer door could fog up, obscuring the cameras’ view.

In complex situations like these, the new model can quickly and accurately determine the actual items taken by each shopper.

This is thanks to its ability to simultaneously process inputs from various sources (including, in this scenario, weight sensors on the fridge shelves), continuously learn from these inputs, and decide which are most important in order to accurately sort out who took what.

This minimises receipt delays and increases the ease of deployment of the technology for retailers.

The new Just Walk Out AI model reduces the rare need for model retraining in unfamiliar shopping scenarios, as the self-learning system continues to teach itself.

To support Just Walk Out’s continuous learning, it is trained on a 3D map of the store to understand placement of fixtures (such as shelves, fridges, freezers, coffee machines, and more) and products, as well as an image catalog of the store’s merchandise so it can visually recognise every item accurately.

This enables the new AI system to recognise shopper behaviour, even if the store is remerchandised and items are rearranged or misplaced.

Watkins said: “The improvements to our AI system are so seamless that you will continue to enjoy the same contactless checkout-free shopping experience you’ve come to expect at Just Walk Out stores, all while protecting your privacy.”

“Upon entering a Just Walk Out store you present a form of payment at the entry gate. The system immediately associates you with that payment method and begins mapping your journey around the store, adding or subtracting items from your virtual cart as you pick them up or put them down.”

“Just Walk Out does not collect any biometric information - the system only tracks how you interact with the products and fixtures (such as shelves or fridges), correctly identifying the products and quantities you leave the store with.”

Just Walk Out is currently available in over 170 third-party locations at the likes of airports, stadiums, universities, hospitals, in the US, UK, Australia, and Canada.

Amazon says it will launch more Just Walk Out stores in 2024 than any year prior, more than doubling the number of third-party stores with the technology this year.

Watkins concluded: “As we scale, the system will continue to learn from everyday shopping scenarios and raise the bar for accuracy and convenience, delivering the benefits of AI to retailers and customers around the world.”

2024 RTIH INNOVATION AWARDS

Checkout-free stores are a key focus area for the sixth edition of the RTIH Innovation Awards, which is now open for entries.

The awards, sponsored by CADS, 3D Cloud, and Retail Technology Show 2025, celebrate global tech innovation in a fast moving omnichannel world.

It’s free to enter and you can do so across multiple categories.

Key 2024 dates

Friday, 25th October: Award entry deadline 

Tuesday, 29th October: 2024 shortlist revealed

30th October-6th November: Judging days

Thursday, 21st November: Winners announced at the 2024 RTIH Innovation Awards ceremony, to be held at RIBA’s 66 Portland Place HQ in Central London.