I couldn't tell you if our matching actually worked

Sri R Chief Janitor

For a good few months I could not have answered that in a way I trusted. Not for lack of numbers, we had plenty, similarity scores, confidence scores, velocity, all the usual internal numbers. But a confidence score is the model telling you how sure it is, which is a different thing from it being right. None of those numbers had ever been checked against an actual answer: a person, or something standing in for one, saying yes, that product fits the trend, or no, it doesn't. So I had a dashboard full of numbers and no real idea whether the matching was any good. And I kept coming back to the same thing: if I can't tell good from bad, I can't tell whether the changes we keep shipping are helping or quietly making it worse. So I stopped adding things and went to work out how you actually grade this.

Working out how to grade it

What I needed was the kind of call a fashion expert makes without thinking: look at a product against a trend and say whether it belongs. They do it all day, and it's basically what the system is trying to copy. The problem is scale. There are around 50,000 products, and new trends surface every day, so it's never a label-it-once job, every new trend means running that judgement across the whole catalogue again. Nobody keeps pace with that by hand. It's the reason the matching is automated in the first place, and the reason grading it is hard too.

So I went and read up on it. Search evaluation is an old field, mostly figured out, and two ideas from it were the ones I used.

The first is pooling. Rather than grading only the products our own system returns, you gather candidates from a few different methods, tip them all into one pile, and grade the pile. I almost skipped it, and I'm glad I didn't. If you only ever grade what your own system already surfaces, you have no way of seeing the good products it's missing, which is more or less the entire reason you're measuring.

The second is using a model as the grader. I handed each trend-product pair to a strong model, told it to score them the way a fashion expert would, 0 for wrong up to 3 for a perfect fit, and froze those scores as the answer key.

I went back and forth on that last part. Letting an AI mark its own homework felt a bit too convenient, so I went and looked, and it's genuinely an open argument. Some researchers find the models track human judgement well; others warn you can quietly bake the model's own blind spots into the benchmark you're trusting. This isn't the kind of thing you settle quickly, and I didn't want to wave it off by grabbing the first approach off the shelf. There are more careful ways to do it, so that's what I used: pool from several methods so the grader can't only reward what we already do, keep the scores frozen and spot-check batches by eye, and treat the result as a rough gauge for whether a change helped rather than proof of anything. It's useful for spotting when something gets worse. I don't read it as the final word on how good the matching is.

There are standard scoring measures, nDCG, recall, a couple of others, and I had to go read what each one meant. They reward putting the right products near the top, and finding as many of the right ones as you can. I was stubborn about one thing: scoring makes no AI calls at all, so it runs in a couple of seconds and I can leave it sitting there as a tripwire every time we change something.

The first score

Then I ran it and the number looked bad. The score for how many of the right products land in the top ten was low enough that I assumed something was broken and went hunting for the bug.

There wasn't one. It took me a while to see what was going on, mostly because I kept assuming something was broken and going back to look for it. Then I actually sat down and counted. A single trend might have twenty-odd products that genuinely fit it, and the score only looks at the top ten. There are more good answers than there are slots to put them in, so a chunk of the right products were always going to fall outside the top ten, however well the thing worked. Once I compared it to what was actually possible, instead of to a perfect score I'd made up in my head, it stopped looking like a failure. We were getting a decent share of the good stuff up top.

A bug this turned up

The measuring turned something up on its own. I don't read the code, I watch what comes out of the system, and what came out kept changing: ask for the same trend twice and the products came back in a slightly different order each time. I didn't know why, and I won't pretend I could have found it by reading anything. It was clearly wrong, so we dug into it. The search runs several queries at once and keeps whichever one finishes first, which is what made the order drift. Small fix, once that was clear. I only noticed at all because I was trying to get the same score twice and couldn't.

Trying to make it better

The ranking was okay, not good, and I didn't know how to make it better, so I read more. The same idea kept coming up. The fast search we use looks at a trend and a product separately and asks whether they're roughly in the same territory. That's quick, and it's enough to pull a shortlist, but it never actually holds the two up against each other. There's a slower kind of model that does, it takes the trend and one specific product and judges them together. Turns out these are called rerankers. I didn't know that going in.

I didn't know which one to use or where to get one. Other people have already built these and put them out for anyone to use, so we took one of those, nothing built specially for us, and ran it over just the shortlist, a hundred products or so. The scores got better. I can't tell you exactly why it's better, but the shape of it made sense to me once I saw it: it actually looks at the pairing, rather than sizing up the two things apart and hoping they match.

A reranker can only judge on the words it's given about a product, and a lot of our product records are thin, a name, a category, not much else. It helped anyway. It would help more the day those records are richer, and that takes no change to the model at all.

What I learned

The first I already half-knew from the commerce side. Plain keyword matching is useless for trends. "Cottagecore" never appears in a product title that reads "floral midi dress with a pleated hem." The product and the trend rarely share the actual words, even when the product is exactly right, so this has to work on meaning and not on matching text.

The second I didn't expect to care about this much. How a product gets written down ends up limiting everything after it. Every part of this reads that text, the search, the reranker, whatever comes next. A flat one-line title puts a lid on the best model money can buy. So a lot of what looks like a modelling problem is really thin product data. Coming from the commerce side, that one is a little humbling to land on.

There was nothing clever about any of it. It was reading, getting things wrong, and refusing to ship changes I couldn't check. But once I could measure it, everything after that went faster, and it's the order I'll keep working in: build the way to measure something before I try to improve it.

Look at the references below and the dates stand out, 1979, 1993, 1999. Nothing I leaned on was new. Pooling, the scoring measures, all of it was worked out decades ago, in other fields, for problems that had nothing to do with fashion. I keep running into that. The answers are mostly already out there, sitting in some paper from thirty years ago. What takes the time is working on your own problem closely enough to see which old idea it maps onto, and then connecting the two.