We trained the robot's eyes before building the robot

Our Retail Lab is working on a store where nothing sits on an open shelf. You order on your phone, a robot picks your items out of totes on a rotating carousel, and a sealed box comes out through a window.

Before anyone spends money on a robot arm, there is a cheaper question to answer. When the order says "one tin, one carton", can a camera looking into a tote find them? So we built the store in software first and asked it.

A store made of pixels

The store already existed as a 3D model, the one on the Retail Lab page. We took three products out of it: a tin, a carton and a 32 cm cleaner jug. Then we put them into a simulated tote under a simulated camera.

The simulator is NVIDIA Isaac Sim, running on a single graphics card in our office. Each picture is built from scratch. The simulator drops one to four products into the tote, upright or on their side, turned any which way, never overlapping and never through a wall. Then it picks a camera height, a lighting temperature somewhere between a warm and a neutral bulb, and a tote colour, and renders.

The useful part is what comes with every picture. The simulator knows exactly where it put everything, so each image arrives already labelled: a box around every product, a mask of every pixel it covers, and the distance to each one. Nobody draws a single box by hand.

We made 4,000 pictures to learn from and 600 more, from a different random seed, to test on. The machine produced about 190 a minute.

First, can an arm even reach?

A camera is pointless if the arm cannot get there. On the Retail Lab page we already admitted that the arm in our original design needed 2.65 metres of reach, and nothing in its class sold today comes close.

So we checked a standard, widely sold arm, the Universal Robots UR5e, using the maker's own published geometry. Pointing straight down, it reaches about 0.9 metres. If the arm's column sits right at the edge of the carousel, the farthest product in the tote is 0.81 metres away. It fits.

The order box does not. In the current layout it sits 2.26 metres from that spot. The answer is to bring the conveyor to the arm, not to buy a bigger arm. That is a drawing change, not a budget change, and we would rather find it now than after the order is placed.

The score

We trained an off-the-shelf detection model for 19 minutes. On the 600 test pictures it found every product, got the type right every time, and took under 2 milliseconds per picture.

A perfect score in a first experiment usually means the experiment was too easy. It was.

Why we do not trust it yet

Every product in the test set was at least 80% visible. Nothing was stacked, nothing leaned against a wall at an angle, and the camera always looked almost straight down. The model never saw the hard case, so it never had a chance to fail on it.

There is still something real in the result. Working out which product is in a tote, from a known list and a fixed camera, is not the hard part of this machine. That is a design choice paying off. Camera checkout systems had to work out what a crowd of shoppers did with thousands of products on open shelves. We only have to confirm what we picked, from a list we control.

The bug that passed every check

The first version of the generator had a flaw. Products from earlier pictures were never cleared away, so they piled up in later ones. The pictures looked fine, the labels were accurate for what was drawn, and our automatic quality checks passed.

We caught it only by comparing each picture's labels against the simulator's own record of what it had meant to place. One picture that should have held two products had five. After the fix, all 4,000 matched.

That is the lesson we would pass on to anyone doing this. A synthetic dataset can be wrong in a way that looks exactly right. Check it against what you meant, not just against itself.

What comes next

• Harder scenes. Stacked products, clutter, items tipped against the wall, the camera at an angle. We keep going until the score stops being perfect, because that is where the useful information starts.
• Where to grip, not just what it is. Our simulation already showed that the 32 cm jug, gripped in the middle, cannot clear a 30 cm tote rim. The next version labels the right place to grip and whether the item will clear, so a model can learn it.
• Real photographs. We will buy the three real products and photograph 200 real totes. Every number above is simulated and will get worse against real pictures. That real number is the only one worth putting in front of an investor.

A simulated store is cheap to be wrong in. That is why we are being wrong in it first.