How Amazon's line-less grocery service might really work
SAN FRANCISCO — The Amazon Go grocery store, now in the testing stage in Seattle, sounds like a dream come true for holiday shoppers waiting in long lines.
The underlying technology seems to be routed in terra firma, however, a mix of cameras, microphones and the massive servers that Amazon uses to run its cloud computing service and power digital assistant Alexa.
The concept promises to let shoppers walk into a store, pick things up, and walk out, thereby skipping the checkout line while everything acquired gets automatically charged to a credit card.
Amazon's explanation on how it works in its video is heavy on buzzwords: computer vision, deep learning algorithms and sensor fusion.
The company declined to comment on the technology behind Amazon Go, but a patent filed by the company in 2014 and published in 2015 may shed some light on the process.
It appears to rely on cameras and microphones — lots and lots of them. The tech is similar to what's used to allow self-driving cars to navigate the world.
According to the patent, each customer entering the store would be tagged as they entered. In Amazon's video, they tag their smartphone, which contains the Amazon Go app, as they walk in.
That then allows the store's surveillance system to identify the customer so that it can track them as they move throughout the space. Cameras pick up images of when they stop in front of shelves, what items they picked up and whether the item stayed in their hand or went back to the shelf.
As the patent puts it, "when the user's hand is removed from the inventory location, one or more images may be captured of the user's hand as it exits the inventory location. Those images may be compared to determine whether a user has picked an item from the inventory location or placed an item in the inventory location."
As in, "image analysis may be performed on the first image to determine a skin tone color of the user's hand and pixels including that color, or a range of colors similar to the identified skin tone color, may be identified to represent the user's hand."
That could potentially be used to distinguish between two people each reaching for things on adjacent shelves, as skin tone is very individual.
Reverse echolocation
This is all hugely, even astoundingly, computing intensive. But it's also exactly what Amazon has become very good at with its Alexa voice-control system. That system sends sound files of spoken commands up to Amazon's cloud computing network, where they are identified, turned into digital commands, answered and then sent back down to the Echo device.
In fact, some of that expertise may be explicit in the Amazon Go system, as microphones are also used in the stores to determine where users are by the noises that they make. It even tracks them by noticing the time difference between the audio signals received by each microphone in the store, a kind of reverse echolocation.
The system also uses infrared, pressure and load sensors on the shelves that note when items have been picked up and whether they are put back. These sensors also feed into the store's continuous sense of where everything, and everyone, in it are at any moment.
Amazon just opened a grocery store without a checkout line
“The challenge is whether you can make it cost-effective. Has Amazon come up with the secret sauce?” said Forrester analyst Brendan Witcher.
Many in the industry had thought something like this would happen through the use of tiny RFID, or radio-frequency ID, tags. These send out a constant “I am here!” message, allowing the system to track whatever they’re attached to.
“They’ve come down in price, they’re now only about five cents each. But it’s still pretty labor intensive to RFID tag everything,” said Witcher.
It's been suggested for years. In fact, IBM released an ad about a store where you just picked things up and walked out back in 2006, he said.
But looking at the patent, it seems as if Amazon’s found a less expensive, more technologically intensive, way to make that work.
The power of information
Knowing who a customer is from the moment they walk through the door, and then tracking everything they do once inside, could be hugely useful to a company, said David Hewitt, an analyst with the consulting firm SapientRazorfish.
“Think about turning on a digital display that markets to the person who’s in the aisle right then. I know it’s a little Big Brother, but all this data is welcome if it makes the shopping process better,” he said.
Amazon could also license the technology to other supermarkets. That would be a win-win for Amazon, because it would be selling not only the technology but also the cloud computing power of its AWS cloud services.
In the end, whether this is economically viable may not matter as much to Amazon as it would to another company, said Witcher.
Investors have been happy to let Amazon lose money quarter after quarter as long as it seems to be moving forward in its technological abilities.
“The market’s given them free rein to fail. They’re allowed to try interesting — and expensive, things,” he said.
And even if the technology is too pricey to actually implement, Amazon will have done what it does best: “Learn and then apply that new knowledge to whatever makes the most sense,” Witcher said.