As a Developer Advocate in the IoT and Edge AI ecosystem for many years now, I have seen lots of interesting, unique, and innovative products and projects get built, covering all sorts of use-cases. However, one use-case I had seen mentioned, but never actually witnessed, was retail inventory counting using computer vision. I knew that there were startups trying to tackle this (very) important process, and sure enough, a few weeks ago I saw my first inventory robot making its way up and down the aisles at my local grocery store. It was not this exact unit, but was similar in concept:

These types of autonomous robots are essentially a camera system housed in injection molded plastic, with a compute platform embedded in the base and a small drive system for navigation and propulsion, plus a battery and charging circuitry. It has connectivity to the internet in order to transmit scanning results and update inventory in real-time, as well. But as someone with a conceptual understanding of how these systems work, I couldn’t stop thinking about two critical factors that need to be solved in this type of deployment.

My first thought was, “What happens when a supplier changes the look of an item?” This is a common occurrence in retail, with labels and branding changing, limited time releases, special editions, and other marketing reasons for changing the appearance of a product on the shelf (even if the actual recipe or food item remains identical). As a simple example, take a look at this updated branding on Cherry Coke:

If a computer vision model is trained on the labeling of the “original” Cherry Coke on the left, and that model is highly accurate, but does not generalize well, then the new labeling shown on the right may not be identified by the camera system. When the distributor sends the next shipment of Cherry Coke, and the staff stocks the shelves with the new bottles, the inventory robot may not be able to identify the Cherry Coke, and falsely order new shipments - when in fact there is plenty of Cherry Coke on the shelf.

With thousands of items in a typical grocery store, and constant rebranding and relabeling of items by manufacturers, this can quickly deem the robot ineffective. So there must be a way to update the robot and send newly trained object detection models to the compute system onboard.

However, a second concern now came to mind: “What about security?” There are actually two distinct processes to consider here, covering both the booting of the robot and then the updating of the products. Knowing that the robot is roaming around a store with thousands of shoppers per day, it would not be far-fetched to believe that some highly-skilled patrons might get a bit “curious” and try to access the onboard compute systems.

Threat vectors in the form of social engineering (attacker posing as a maintenance technician in order to “work on” the robot), or simply attempting to access the robot via wireless connectivity, Bluetooth, etc. would not be uncommon occurrences. Thus, the robot manufacturer needs to implement secure boot processes with TPMs or secure enclaves that ensure the boot process is not tampered with, and that the software stack running is the correct and verified application portfolio. Beyond that, when it is time to update the AI models on the robot, again SSL/TLS encryption and only validated and verified sources should be allowed to write the new files to the robot. There is one additional consideration though. Large grocery chains own hundreds or thousands of stores.The secure delivery of AI model updates needs to happen over-the-air at scale, to the entire fleet of robots.

A Demo is Needed

With all of this context and background story now explained, here is the real reason I spent so much time pondering all of this…I needed to build a demo for an upcoming conference, and this was a perfect use-case to attempt to replicate (at a smaller scale)!

I didn’t necessarily need the entire robot system, with autonomous navigation and the backend inventory management system, I simply wanted to demonstrate the Edge AI pieces, but I needed to find a secure and reliable method to ensure verified boot and secure delivery of updated AI models over-the-air.

I also needed to find a product that underwent a packaging or labeling update, but that ultimately the same product…and purchase both iterations in order to show the update process live in-person.

So, I did a bit of research, found my solution, and then headed back to the grocery store.

Thistle Technologies

Starting with the easier of the two tasks, the research phase, I had already come across Thistle Technologies in the past. I had seen them a while back in some media coverage, and recalled what they were trying to accomplish. I needed a quick refresher, but sure enough, Thistle is solving this exact problem of secured boot and updates in embedded products and solutions. From the Thistle website, I could see right away what they are focused on: “Easily Ship Validated Updates With signed updates and reliable fail-over, devices can be updated with confidence.”

Additionally, scrolling down to the bottom, I noticed “Thistle Technologies set of tools allowing device manufacturers to cryptographically ensure all software running on their device fleet has come from an approved source.”

So that checks both of my boxes, and even better, I see right on the website how easy this will be to deploy my updated AI models that I will build:

In the Thistle Documentation, I noticed there are quick-start instructions for a Raspberry Pi and for a pair of BeagleBone devices. There are also quick-starts for File Updates, and for Thistle Verified Boot. I was able to run through these successfully using a Raspberry Pi 4 simply by following the documentation, and came away with a Thistle Device that registered and checked-in, that I could then see in my Dashboard. Because I knew this demo would be used for enterprise conversations however, I then took the Raspberry Pi offline and instead switched to a Texas Instruments SK-AM68A, which is an industrial SoM and carrier board targeted at product development, as it has long-term availability, and the SoM can be used on customized carriers purpose-built for a device. This gave me enterprise credibility, and also a great talking point about the journey from prototype to production.

Once this TI SK-AM68A was up and running with the Thistle Update Client (TUC), and I could see it registered in the Dashboard, it was time to test my theory of building a computer vision model trained on a product label, that could not identify the same product with it’s new branding. On my return trip to the grocery store, I could not find any bottles of Cherry Coke with old and new labeling, but I did stumble upon a pair of wine bottles that fit the description! Sure enough, side-by-side on a shelf were two different brandings of “The Federalist” red wine. Checking the UPC Bar Code, they had the exact same item number, ultimately meaning these two different-looking wines were indeed the same wine as far as the store was concerned. In fact, when I went to pay, the cashier scanned each bottle but was puzzled and believed there was an error with one of them, as they came up the same on his register, but he noticed they looked different. I explained to him exactly what was going on, and away I went.

Back home, I used Edge Impulse to train an object detection model on the first label, by taking pictures of the bottle from a variety of angles, and with differing light and backgrounds. You can learn more about building a computer vision project in Edge Impulse here. After training was complete, I deployed the model to the SK-AM68A, and sure enough it was able to identify the label and place a bounding box around the object of interest in the camera frame.

Placing the second bottle in the field of view of the camera did not produce a second bounding box, as I had theorized would be the case. At this point, I had replicated my problem statement, so it was time to then create the solution.

First, I needed to create a new object detection model, this time trained on the updated label. I started a new project in Edge Impulse, and repeated the same steps: Collect a wide variety of images of the new bottle, draw bounding boxes around the point of interest in the image (this is called “Labeling”), and then train the model. At the end of the process, I now had a discrete model trained only on the new label, so I needed to replace the running model on the board.

Thistle Update Client for Secure Over-the-Air Deployments

Now that I had two models built, I needed a way to securely push each one individually, in order to replicate the functionality of what would occur on the inventory robot. Thistle made this easy, with both individual file and folder updates, or full partition updates available. Full partition updates are very important for security purposes when updating core operating system components, in order to address vulnerabilities and exploits as they are discovered and patched. Thistle also supports fallback booting, so if the partition update does not complete successfully and the device fails to boot up into the new OS, it will revert to the previous partition so that developers and engineers can investigate what went wrong — but not lose access to the device out in the field.

In this case, as I was only interested in updating a single AI model, I chose the File Update method. There are two ways to perform this update, either via the Thistle Update Client running on the device, or by using the web interface to select your files and send them directly to the device. I ended up writing a small bash script that would kill any active Edge Impulse Model Runner, then relaunch the Runner with the new model, so in my case it was a bit easier to simply use TUC via the CLI, but performing this type of update via the Thistle Dashboard is as easy as it gets!

With this framework in place, I could then push the first AI model to the TI SK-AM68A, and my first wine bottle was identified and a bounding box was placed around the label, with the second bottle left unrecognized. Because I was working via the command line, I used the Thistle Release Helper to then package up the second model and distribute it to the device. Watching the feed from the camera, once the model was securely downloaded and the Runner restarted, the bounding box in the camera stream jumped over to the second bottle — proving the concept worked!

Edge AI Updates, Delivered Securely

While purchasing an inventory robot could have also made for an interesting demo at the conference, this smaller-scale prototype worked quite well, and was much more cost effective. However, the concept was proven, from both the problem and solution perspective. Thistle was able to address both the secure and verified boot process, as well as the secure delivery of newly trained AI models that account for updated labeling and branding of retail items. This system could be used any time a manufacturer changes the appearance of an item, ensuring an inventory robot is continually updated and its computer vision system can adapt to changing conditions once it’s deployed to a store.

If you have any questions or comments, be sure to reach out.